[doc](fix) cold hot separation doc fix (#19757)
This commit is contained in:
catpineapple
@ -104,12 +104,12 @@ For details, please refer to the [resource](../sql-manual/sql-reference/Data-Def
|
||||
|
||||
### Some restrictions
|
||||
|
||||
- A single table or a single partition can only be associated with one storage policy. After association, the storage policy cannot be dropped
|
||||
- A single table or a single partition can only be associated with one storage policy. After association, the storage policy cannot be dropped，need to solve the relationship between the two.
|
||||
- The object information associated with the storage policy does not support modifying the data storage path information, such as bucket, endpoint, and root_ Path and other information
|
||||
- Currently, the storage policy only supports creation and modification, not deletion
|
||||
- Currently, the storage policy only supports creation,modification and deletion. before deleting, you need to ensure that no table uses this storage policy.
|
||||
|
||||
## Show size of objects occupied by cold data
|
||||
1. Through show proc '/backends', you can view the size of each object being uploaded to, and the RemoteUsedCapacity item.
|
||||
1. Through show proc '/backends', you can view the size of each object being uploaded to, and the RemoteUsedCapacity item.There is a slight delay in this method
|
||||
|
||||
2. Through show tables from tableName, you can view the object size occupied by each table, and the RemoteDataSize item.
|
||||
|
||||
@ -122,7 +122,34 @@ As above, cold data introduces the cache in order to optimize query performance.
|
||||
- The be parameter `file_cache_max_size_per_disk` can set the disk size occupied by the cache. Once this setting is exceeded, the cache that has not been accessed for the longest time will be deleted. The default is 0, means no limit to the size, unit: byte.
|
||||
- The be parameter `file_cache_type` is optional `sub_file_cache` (segment the remote file for local caching) and `whole_file_cache` (the entire remote file for local caching), the default is "", means no file is cached, please set it when caching is required this parameter.
|
||||
|
||||
|
||||
|
||||
## cold data compaction
|
||||
The time when cold data is imported is from the moment when the data rowset file is written to the local disk, plus the cooling time. Since the data is not written and cooled at one time, to avoid the problem of small files in the object storage, doris will also perform compaction of cold data.
|
||||
However, the frequency of cold data compaction and the priority of resource occupation are not very high. Specifically, it can be adjusted by the following be parameters:
|
||||
- The be parameter `cold_data_compaction_thread_num` can set the concurrency of executing cold data compaction, the default is 2.
|
||||
- The be parameter `cold_data_compaction_interval_sec` can set the time interval for executing cold data compaction, the default is 1800, unit: second, that is, half an hour.
|
||||
|
||||
|
||||
## cold data schema change
|
||||
The supported schema change types after data cooling are as follows:
|
||||
* Add and delete columns
|
||||
* Modify column type
|
||||
* Adjust column order
|
||||
* Add and modify Bloom Filter
|
||||
* Add and delete bitmap index
|
||||
|
||||
## cold data Garbage collection
|
||||
The garbage data of cold data refers to the data that is not used by any Replica. Object storage may have garbage data generated by the following situations:
|
||||
1. Failed to upload rowset but uploaded some segments successfully.
|
||||
2. After the FE re-selects the CooldownReplica, the rowset versions of the old and new CooldownReplicas are inconsistent, and the FollowerReplicas synchronize the CooldownMeta of the new CooldownReplica. The rowsets with inconsistent versions in the old CooldownReplica are not used by the Replica and become garbage data.
|
||||
3. After the cold data compaction, the rowset before the merge cannot be deleted immediately because it may be used by other Replicas, but in the end FollowerReplicas all use the latest merged rowset, and the rowset before the merge becomes garbage data.
|
||||
|
||||
In addition, the garbage data on the object will not be cleaned up immediately.
|
||||
The be parameter `remove_unused_remote_files_interval_sec` can set the garbage collection interval of cold data, the default is 21600, unit: second, that is, 6 hours.
|
||||
|
||||
|
||||
## Unfinished Matters
|
||||
|
||||
- After the data is frozen, there are new data updates or imports, etc. The compression has not been processed at present.
|
||||
- The schema change operation after the data is frozen is not supported at present.
|
||||
- Currently, there is no way to query the tables associated with a specific storage policy.
|
||||
- The acquisition of some remote occupancy indicators is not perfect enough.
|
||||
@ -103,16 +103,16 @@ ALTER TABLE create_table_partition MODIFY PARTITION (*) SET("storage_policy"="te
|
||||
|
||||
### 一些限制
|
||||
|
||||
- 单表或单partition只能关联一个storage policy，关联后不能drop掉storage policy
|
||||
- 单表或单partition只能关联一个storage policy，关联后不能drop掉storage policy，需要先解二者的除关联。
|
||||
- storage policy关联的对象信息不支持修改数据存储path的信息，比如bucket、endpoint、root_path等信息
|
||||
- storage policy目前只支持创建和修改，不支持删除
|
||||
- storage policy支持创建和修改和支持删除，删除前需要先保证没有表引用此storage policy。
|
||||
|
||||
## 冷数据占用对象大小
|
||||
方式一：
|
||||
通过show proc '/backends'可以查看到每个be上传到对象的大小，RemoteUsedCapacity项
|
||||
通过show proc '/backends'可以查看到每个be上传到对象的大小，RemoteUsedCapacity项，此方式略有延迟。
|
||||
|
||||
方式二：
|
||||
通过show tablets from tableName可以查看到表的每个tablet占用的对象大小，RemoteDataSize项
|
||||
通过show tablets from tableName可以查看到表的每个tablet占用的对象大小，RemoteDataSize项。
|
||||
|
||||
## 冷数据的cache
|
||||
上文提到冷数据为了优化查询的性能和对象存储资源节省，引入了cache的概念。在冷却后首次命中，Doris会将已经冷却的数据又重新加载到be的本地磁盘，cache有以下特性：
|
||||
@ -122,8 +122,31 @@ ALTER TABLE create_table_partition MODIFY PARTITION (*) SET("storage_policy"="te
|
||||
- be参数`file_cache_max_size_per_disk` 可以设置cache占用磁盘大小，一旦超过这个设置，会删除最久未访问cache，默认是0，单位：字节，即不限制大小。
|
||||
- be参数`file_cache_type` 可选项`sub_file_cache`（切分远端文件进行本地缓存）和`whole_file_cache`（整个远端文件进行本地缓存），默认为""，即不缓存文件，需要缓存的时候请设置此参数。
|
||||
|
||||
## 冷数据的compaction
|
||||
冷数据传入的时间是数据rowset文件写入本地磁盘时刻起，加上冷却时间。由于数据并不是一次性写入和冷却的，因此避免在对象存储内的小文件问题，doris也会进行冷数据的compaction。
|
||||
但是，冷数据的compaction的频次和资源占用的优先级并不是很高。具体可以通过以下be参数调整：
|
||||
- be参数`cold_data_compaction_thread_num`可以设置执行冷数据的compaction的并发，默认是2。
|
||||
- be参数`cold_data_compaction_interval_sec` 可以设置执行冷数据的compaction的时间间隔，默认是1800，单位：秒，即半个小时。
|
||||
|
||||
## 冷数据的schema change
|
||||
数据冷却后支持schema change类型如下：
|
||||
- 增加、删除列
|
||||
- 修改列类型
|
||||
- 调整列顺序
|
||||
- 增加、修改 Bloom Filter
|
||||
- 增加、删除 bitmap index
|
||||
|
||||
## 冷数据的垃圾回收
|
||||
冷数据的垃圾数据是指没有被任何Replica使用的数据，对象存储上可能会有如下情况产生的垃圾数据：
|
||||
1. 上传rowset失败但是有部分segment上传成功。
|
||||
2. FE重新选CooldownReplica后，新旧CooldownReplica的rowset version不一致，FollowerReplica都去同步新CooldownReplica的CooldownMeta，旧CooldownReplica中version不一致的rowset没有Replica使用成为垃圾数据。
|
||||
3. 冷数据Compaction后，合并前的rowset因为还可能被其他Replica使用不能立即删除，但是最终FollowerReplica都使用了最新的合并后的rowset，合并前的rowset成为垃圾数据。
|
||||
|
||||
另外，对象上的垃圾数据并不会立即清理掉。
|
||||
be参数`remove_unused_remote_files_interval_sec` 可以设置冷数据的垃圾回收的时间间隔，默认是21600，单位：秒，即6个小时。
|
||||
|
||||
|
||||
## 未尽事项
|
||||
|
||||
- 数据被cooldown后，又有新数据update或导入等，compaction目前没有处理
|
||||
- 数据被cooldown后，schema change操作，目前不支持
|
||||
- 目前暂无方式查询特定storage policy 关联的表。
|
||||
- 一些远端占用指标更新获取不够完善
|
||||
|
||||
Reference in New Issue
Block a user