database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[doc](file-cache) fix wrong config (#29156)

Browse Source 
* [doc](file-cache) fix wrong config

* 2
This commit is contained in:
Mingyu Chen
2023-12-27 18:51:01 +08:00
committed by GitHub
parent d70743a2fc
commit 286af3bce9
2 changed files with 55 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
docs/en/docs/lakehouse/filecache.md
View File

@ -53,42 +53,43 @@ SET GLOBAL enable_file_cache = true;
> The File Cache is only applicable to external queries for files (such as Hive, Hudi). It has no effect on internal table queries, or non-file external queries (such as JDBC, Elasticsearch), etc.
### Configurations for BE
Add settings to the BE node's configuration file `conf/be.conf`, and restart the BE node for the configuration to take effect.
| Parameter | Description |
| --- | --- |
| `enable_file_cache` | Whether to enable File Cache, default false |
| `file_cache_path` | Parameters about cache path, json format, for exmaple: `[{"path": "/path/to/file_cache1", "total_size":53687091200,"query_limit": "10737418240"},{"path": "/path/to/file_cache2", "total_size":53687091200,"query_limit": "10737418240"},{"path": "/path/to/file_cache3", "total_size":53687091200,"query_limit": "10737418240"}]`. `path` is the path to save cached data; `total_size` is the max size of cached data; `query_limit` is the max size of cached data for a single query. |
| `file_cache_min_file_segment_size` | Min size of a single cached block, default 1MB, should greater than 4096 |
| `file_cache_max_file_segment_size` | Max size of a single cached block, default 4MB, should greater than 4096 |
| `enable_file_cache_query_limit` | Whether to limit the cache size used by a single query, default false |
| `clear_file_cache` | Whether to delete the previous cache data when the BE restarts, default false |
| Parameter | Required | Description |
| --- | --- | --- |
| `enable_file_cache` | Yes | Whether to enable File Cache, default false |
| `file_cache_path` | Yes | Parameters about cache path, json format, for exmaple: `[{"path": "/path/to/file_cache1", "total_size":53687091200,"query_limit": 10737418240},{"path": "/path/to/file_cache2", "total_size":53687091200,"query_limit": 10737418240},{"path": "/path/to/file_cache3", "total_size":53687091200,"query_limit": 10737418240}]`. `path` is the path to save cached data; `total_size` is the max size of cached data; `query_limit` is the max size of cached data for a single query. |
| `file_cache_min_file_segment_size` | No | Min size of a single cached block, default 1MB, should greater than 4096 |
| `file_cache_max_file_segment_size` | No | Max size of a single cached block, default 4MB, should greater than 4096 |
| `enable_file_cache_query_limit` | No | Whether to limit the cache size used by a single query, default false |
| `clear_file_cache` | No | Whether to delete the previous cache data when the BE restarts, default false |
## Check whether a query hits cache
Execute `set enable_profile = true` to enable the session variable, and you can view the query profile in the Queris tab of FE's web page. The metrics related to File Cache are as follows:
```
- FileCache:
- IOHitCacheNum: 552
- IOTotalNum: 835
- ReadFromFileCacheBytes: 19.98 MB
- ReadFromWriteCacheBytes: 0.00
- ReadTotalBytes: 29.52 MB
- WriteInFileCacheBytes: 915.77 MB
- WriteInFileCacheNum: 283
- FileCache: 0ns
- BytesScannedFromCache: 2.02 GB
- BytesScannedFromRemote: 0.00
- BytesWriteIntoCache: 0.00
- LocalIOUseTimer: 2s723ms
- NumLocalIOTotal: 444
- NumRemoteIOTotal: 0
- NumSkipCacheIOTotal: 0
- RemoteIOUseTimer: 0ns
- WriteCacheIOUseTimer: 0ns
```
- `IOTotalNum`: Number of remote access
- `IOHitCacheNum`: Number of cache hits
- `ReadFromFileCacheBytes`: Amount of data read from cache file
- `ReadTotalBytes`: Total amount of data read
- `SkipCacheBytes`: Failed to create the cache file, or the cache file was deleted. The amount of data that needs to be read from the remote again
- `WriteInFileCacheBytes`: Amount of data saved to cache file
- `WriteInFileCacheNum`: The number of blocks saved, so 'WriteInFileCacheBytes'/' WriteInFileCacheBytes' is the average size of blocks
- `BytesScannedFromCache`: The amount of data read from the local cache.
- `BytesScannedFromRemote`: The amount of data read from the remote end.
- `BytesWriteIntoCache`: The amount of data written to the cache.
- `LocalIOUseTimer`: Local cached IO time.
- `RemoteIOUseTimer`: IO time for remote reading.
- `NumLocalIOTotal`: The number of locally cached IOs.
- `NumRemoteIOTotal`: Number of remote IOs.
- `WriteCacheIOUseTimer`: IO time to write cache.
`IOHitCacheNum` / `IOTotalNum` Equal to 1, indicating that read data only from file cache
If `BytesScannedFromRemote` is 0, it means all caches are hit.
`ReadFromFileCacheBytes` / `ReadTotalBytes` Equal to 1, indicating that read data only from file cache
`ReadFromFileCacheBytes` The smaller the better, the smaller the amount of data read from remote

53
docs/zh-CN/docs/lakehouse/filecache.md
View File

@ -57,39 +57,40 @@ SET GLOBAL enable_file_cache = true;
添加参数到 BE 节点的配置文件 conf/be.conf 中，并重启 BE 节点让配置生效。
| 参数 | 说明 |
| --- | --- |
| `enable_file_cache` | 是否启用 File Cache，默认 false |
| `file_cache_path` | 缓存目录的相关配置，json格式，例子: `[{"path": "/path/to/file_cache1", "total_size":53687091200,"query_limit": "10737418240"},{"path": "/path/to/file_cache2", "total_size":53687091200,"query_limit": "10737418240"},{"path": "/path/to/file_cache3", "total_size":53687091200,"query_limit": "10737418240"}]`。`path` 是缓存的保存路径，`total_size` 是缓存的大小上限，`query_limit` 是单个查询能够使用的最大缓存大小。 |
| `file_cache_min_file_segment_size` | 单个 Block 的大小下限，默认 1MB，需要大于 4096 |
| `file_cache_max_file_segment_size` | 单个 Block 的大小上限，默认 4MB，需要大于 4096 |
| `enable_file_cache_query_limit` | 是否限制单个 query 使用的缓存大小，默认 false |
| `clear_file_cache` | BE 重启时是否删除之前的缓存数据，默认 false |
| 参数 | 必选项 | 说明 |
| --- | --- | --- |
| `enable_file_cache` | 是 | 是否启用 File Cache，默认 false |
| `file_cache_path` | 是 | 缓存目录的相关配置，json格式，例子: `[{"path": "/path/to/file_cache1", "total_size":53687091200,"query_limit": 10737418240},{"path": "/path/to/file_cache2", "total_size":53687091200,"query_limit": 10737418240},{"path": "/path/to/file_cache3", "total_size":53687091200,"query_limit": 10737418240}]`。`path` 是缓存的保存路径，`total_size` 是缓存的大小上限，`query_limit` 是单个查询能够使用的最大缓存大小。 |
| `file_cache_min_file_segment_size` | 否 | 单个 Block 的大小下限，默认 1MB，需要大于 4096 |
| `file_cache_max_file_segment_size` | 否 | 单个 Block 的大小上限，默认 4MB，需要大于 4096 |
| `enable_file_cache_query_limit` | 否 | 是否限制单个 query 使用的缓存大小，默认 false |
| `clear_file_cache` | 否 | BE 重启时是否删除之前的缓存数据，默认 false |
### 查看 File Cache 命中情况
执行 `set enable_profile=true` 打开会话变量，可以在 FE 的 web 页面的 Queris 标签中查看到作业的 Profile。File Cache 相关的指标如下:
```
- FileCache:
- IOHitCacheNum: 552
- IOTotalNum: 835
- ReadFromFileCacheBytes: 19.98 MB
- ReadFromWriteCacheBytes: 0.00
- ReadTotalBytes: 29.52 MB
- WriteInFileCacheBytes: 915.77 MB
- WriteInFileCacheNum: 283
- FileCache: 0ns
- BytesScannedFromCache: 2.02 GB
- BytesScannedFromRemote: 0.00
- BytesWriteIntoCache: 0.00
- LocalIOUseTimer: 2s723ms
- NumLocalIOTotal: 444
- NumRemoteIOTotal: 0
- NumSkipCacheIOTotal: 0
- RemoteIOUseTimer: 0ns
- WriteCacheIOUseTimer: 0ns
```
- `IOTotalNum`: 远程访问的次数
- `IOHitCacheNum`: 命中缓存的次数
- `ReadFromFileCacheBytes`: 从缓存文件中读取的数据量
- `ReadTotalBytes`: 总共读取的数据量
- `SkipCacheBytes`: 创建缓存文件失败，或者缓存文件被删，需要再次从远程读取的数据量
- `WriteInFileCacheBytes`: 保存到缓存文件中的数据量
- `WriteInFileCacheNum`: 保存的 Block 数量，所以 `WriteInFileCacheBytes`/`WriteInFileCacheBytes` 为 Block 的平均大小
- `BytesScannedFromCache`：从本地缓存中读取的数据量。
- `BytesScannedFromRemote`：从远端读取的数据量。
- `BytesWriteIntoCache`：写入缓存的数据量。
- `LocalIOUseTimer`：本地缓存的 IO 时间。
- `RemoteIOUseTimer`：远端读取的 IO 时间。
- `NumLocalIOTotal`：本地缓存的 IO 次数。
- `NumRemoteIOTotal`：远端 IO 次数。
- `WriteCacheIOUseTimer`：写入缓存的 IO 时间。
`IOHitCacheNum` / `IOTotalNum` 等于1，表示缓存完全命中
`ReadFromFileCacheBytes` / `ReadTotalBytes` 等于1，表示缓存完全命中
如果 `BytesScannedFromRemote` 为 0，表示全部命中缓存。