From dbac12bae86b45a063a11844e9a065be3f0a10fe Mon Sep 17 00:00:00 2001 From: Xinyi Zou Date: Wed, 15 Nov 2023 14:02:58 +0800 Subject: [PATCH] [fix](memory)Modify the default conf values of mem_limit and cache_last_version_interval_second (#26945) mem_limit from 80% to 90% cache_last_version_interval_second from 900 to 30 --- be/src/common/config.cpp | 2 +- docs/en/docs/admin-manual/config/be-config.md | 2 +- docs/en/docs/admin-manual/config/fe-config.md | 2 +- docs/en/docs/advanced/cache/partition-cache-manual.md | 6 ++---- docs/en/docs/advanced/cache/query-cache.md | 5 +++-- docs/en/docs/advanced/cache/sql-cache-manual.md | 4 +--- docs/zh-CN/docs/admin-manual/config/be-config.md | 2 +- docs/zh-CN/docs/admin-manual/config/fe-config.md | 2 +- docs/zh-CN/docs/advanced/cache/partition-cache-manual.md | 6 ++---- docs/zh-CN/docs/advanced/cache/query-cache.md | 7 ++++--- docs/zh-CN/docs/advanced/cache/sql-cache-manual.md | 4 +--- .../src/main/java/org/apache/doris/common/Config.java | 2 +- 12 files changed, 19 insertions(+), 25 deletions(-) diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp index 4d58b1fc94..cf202cb393 100644 --- a/be/src/common/config.cpp +++ b/be/src/common/config.cpp @@ -81,7 +81,7 @@ DEFINE_String(memory_mode, "moderate"); // defaults to bytes if no unit is given" // must larger than 0. and if larger than physical memory size, // it will be set to physical memory size. -DEFINE_String(mem_limit, "80%"); +DEFINE_String(mem_limit, "90%"); // Soft memory limit as a fraction of hard memory limit. DEFINE_Double(soft_mem_limit_frac, "0.9"); diff --git a/docs/en/docs/admin-manual/config/be-config.md b/docs/en/docs/admin-manual/config/be-config.md index c11d03345a..d3192d9245 100644 --- a/docs/en/docs/admin-manual/config/be-config.md +++ b/docs/en/docs/admin-manual/config/be-config.md @@ -188,7 +188,7 @@ There are two ways to configure BE configuration items: * Type: string * Description: Limit the percentage of the server's maximum memory used by the BE process. It is used to prevent BE memory from occupying too the machine's memory. This parameter must be greater than 0. When the percentage is greater than 100%, the value will default to 100%. -* Default value: 80% +* Default value: 90% #### `cluster_id` diff --git a/docs/en/docs/admin-manual/config/fe-config.md b/docs/en/docs/admin-manual/config/fe-config.md index 56f77d3f77..a4a85b4908 100644 --- a/docs/en/docs/admin-manual/config/fe-config.md +++ b/docs/en/docs/admin-manual/config/fe-config.md @@ -829,7 +829,7 @@ In order to avoid occupying too much memory, the maximum data size of rows that #### `cache_last_version_interval_second` -Default：900 +Default：30 IsMutable：true diff --git a/docs/en/docs/advanced/cache/partition-cache-manual.md b/docs/en/docs/advanced/cache/partition-cache-manual.md index b88fa65b46..854b5629ea 100644 --- a/docs/en/docs/advanced/cache/partition-cache-manual.md +++ b/docs/en/docs/advanced/cache/partition-cache-manual.md @@ -70,14 +70,12 @@ MySQL [(none)]> set [global] enable_partition_cache=true; If two caching strategies are enabled at the same time, you need to pay attention to the following parameters: ```text -cache_last_version_interval_second=900 +cache_last_version_interval_second=30 ``` If the interval between the latest version of the partition and the present is greater than cache_last_version_interval_second, the entire query result will be cached first. If it is less than this interval, if it meets the conditions of PartitionCache, the PartitionCache data will be pressed. -## Related parameters - -For detailed parameter introduction, see query-cache.md +For detailed parameter introduction and unfinished matters, see query-cache.md. ## Implementation principle example diff --git a/docs/en/docs/advanced/cache/query-cache.md b/docs/en/docs/advanced/cache/query-cache.md index b41b32a801..39811667a9 100644 --- a/docs/en/docs/advanced/cache/query-cache.md +++ b/docs/en/docs/advanced/cache/query-cache.md @@ -111,11 +111,11 @@ cache_result_max_data_size=31457280 3. cache_last_version_interval_second -The minimum time interval between the latest version of the cached query partition and the current version. Only the query results of partitions that are larger than this interval and have not been updated will be cached. The default is 900, in seconds. +The minimum time interval between the latest version of the cached query partition and the current version. Only the query results of partitions that are larger than this interval and have not been updated will be cached. The default is 30, in seconds. ```text vim fe/conf/fe.conf -cache_last_version_interval_second=900 +cache_last_version_interval_second=30 ``` 4. query_cache_max_size_mb and query_cache_elasticity_size @@ -141,6 +141,7 @@ cache_max_partition_count=1024 ## Unfinished business +- SQL contains functions that generate random values, such as random(). Using QueryCache will cause the query results to lose their randomness, and the same results will be obtained every time they are executed. - Can T+1 data also be cached using Partition? Currently it is not supported. - Similar SQL, 2 indicators were queried before, and now 3 indicators are queried. Can the cache of 2 indicators be used? Not currently supported - Partition by date, but the data needs to be summarized by week dimension. Can PartitionCache be used? Not currently supported diff --git a/docs/en/docs/advanced/cache/sql-cache-manual.md b/docs/en/docs/advanced/cache/sql-cache-manual.md index 1966ce68b3..cc33620145 100644 --- a/docs/en/docs/advanced/cache/sql-cache-manual.md +++ b/docs/en/docs/advanced/cache/sql-cache-manual.md @@ -69,6 +69,4 @@ After the first query, if the following three conditions are met, the query resu 3. The query result bytes is less than cache_result_max_data_size in fe.conf. -## Related parameters - -For detailed parameter introduction, see query-cache.md. +For detailed parameter introduction and unfinished matters, see query-cache.md. diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md b/docs/zh-CN/docs/admin-manual/config/be-config.md index 533b7db5f4..3a94883804 100644 --- a/docs/zh-CN/docs/admin-manual/config/be-config.md +++ b/docs/zh-CN/docs/admin-manual/config/be-config.md @@ -197,7 +197,7 @@ BE 重启后该配置将失效。如果想持久化修改结果，使用如下 * 类型：string * 描述：限制BE进程使用服务器最大内存百分比。用于防止BE内存挤占太多的机器内存，该参数必须大于0，当百分大于100%之后，该值会默认为100%。 -* 默认值：80% +* 默认值：90% #### `cluster_id` diff --git a/docs/zh-CN/docs/admin-manual/config/fe-config.md b/docs/zh-CN/docs/admin-manual/config/fe-config.md index a69c13b252..7d1009198a 100644 --- a/docs/zh-CN/docs/admin-manual/config/fe-config.md +++ b/docs/zh-CN/docs/admin-manual/config/fe-config.md @@ -830,7 +830,7 @@ http header size 配置参数 #### `cache_last_version_interval_second` -默认值：900 +默认值：30 是否可以动态配置：true diff --git a/docs/zh-CN/docs/advanced/cache/partition-cache-manual.md b/docs/zh-CN/docs/advanced/cache/partition-cache-manual.md index 0896df49d9..6979606652 100644 --- a/docs/zh-CN/docs/advanced/cache/partition-cache-manual.md +++ b/docs/zh-CN/docs/advanced/cache/partition-cache-manual.md @@ -70,14 +70,12 @@ MySQL [(none)]> set [global] enable_partition_cache=true; 如果同时开启了两个缓存策略，下面的参数，需要注意一下: ```text -cache_last_version_interval_second=900 +cache_last_version_interval_second=30 ``` 如果分区的最新版本的时间离现在的间隔，大于cache_last_version_interval_second，则会优先把整个查询结果缓存。如果小于这个间隔，如果符合PartitionCache的条件，则按PartitionCache数据。 -## 相关参数 - -具体参数介绍见 query-cache.md +具体参数介绍和未尽事项见 query-cache.md。 ## 实现原理示例 diff --git a/docs/zh-CN/docs/advanced/cache/query-cache.md b/docs/zh-CN/docs/advanced/cache/query-cache.md index 5139f2dde5..daaf22800a 100644 --- a/docs/zh-CN/docs/advanced/cache/query-cache.md +++ b/docs/zh-CN/docs/advanced/cache/query-cache.md @@ -111,11 +111,11 @@ cache_result_max_data_size=31457280 3. cache_last_version_interval_second -缓存的查询分区最新版本离现在的最小时间间隔，只有大于这个间隔没有被更新的分区的查询结果才会被缓存，默认 900，单位秒。 +缓存的查询分区最新版本离现在的最小时间间隔，只有大于这个间隔没有被更新的分区的查询结果才会被缓存，默认 30，单位秒。 ```text vim fe/conf/fe.conf -cache_last_version_interval_second=900 +cache_last_version_interval_second=30 ``` 4. query_cache_max_size_mb 和 query_cache_elasticity_size @@ -141,6 +141,7 @@ cache_max_partition_count=1024 ## 未尽事项 -- T+1的数据，是否也可以用Partition缓存? 目前不支持 +- SQL中包含产生随机值的函数，比如 random()，使用 QueryCache 会导致查询结果失去随机性，每次执行将得到相同的结果。 +- T+1的数据，是否也可以用 PartitionCache? 目前不支持 - 类似的SQL，之前查询了2个指标，现在查询3个指标，是否可以利用2个指标的缓存？ 目前不支持 - 按日期分区，但是需要按周维度汇总数据，是否可用PartitionCache？ 目前不支持 diff --git a/docs/zh-CN/docs/advanced/cache/sql-cache-manual.md b/docs/zh-CN/docs/advanced/cache/sql-cache-manual.md index c9e6b68ff7..812329b4cb 100644 --- a/docs/zh-CN/docs/advanced/cache/sql-cache-manual.md +++ b/docs/zh-CN/docs/advanced/cache/sql-cache-manual.md @@ -69,6 +69,4 @@ MySQL [(none)]> set [global] enable_sql_cache=true; 3. 查询结果bytes 小于 fe.conf 中的 cache_result_max_data_size。 -## 相关参数 - -具体参数介绍见 query-cache.md。 +具体参数介绍和未尽事项见 query-cache.md。 diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java index e62b226bf5..b62faee7aa 100644 --- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java +++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java @@ -1231,7 +1231,7 @@ public class Config extends ConfigBase { * This parameter distinguishes between offline and real-time updates */ @ConfField(mutable = true, masterOnly = false) - public static int cache_last_version_interval_second = 900; + public static int cache_last_version_interval_second = 30; /** * Set the maximum number of rows that can be cached