[Enhancement](multi-catalog) Make meta cache batch loading concurrently. (#21471)
I will enhance performance about querying meta cache of hms tables by 2 steps: **Step1** : use concurrent batch loading for meta cache **Step2** : execute some other tasks concurrently as soon as possible **This pr mainly for step1 and it mainly do the following things:** - Create a `CacheBulkLoader` for batch loading - Remove the executor of the previous async cache loader and change the loader's type to `CacheBulkLoader` (We do not set any refresh strategies for LoadingCache, so the previous executor is not useful) - Use a `FixedCacheThreadPool` to replace the `CacheThreadPool` (The previous `CacheThreadPool` just log warn infos and will not throw any exceptions when the pool is full). - Remove parallel streams and use the `CacheBulkLoader` to do batch loadings - Change the value of `max_external_cache_loader_thread_pool_size` to 64, and set the pool size of hms client pool to `max_external_cache_loader_thread_pool_size` - Fix the spelling mistake for `max_hive_table_catch_num`
This commit is contained in:
@ -1694,7 +1694,7 @@ public class Config extends ConfigBase {
|
||||
|
||||
@ConfField(mutable = false, masterOnly = false, description = {"Hive表到分区名列表缓存的最大数量。",
|
||||
"Max cache number of hive table to partition names list."})
|
||||
public static long max_hive_table_catch_num = 1000;
|
||||
public static long max_hive_table_cache_num = 1000;
|
||||
|
||||
@ConfField(mutable = false, masterOnly = false, description = {"获取Hive分区值时候的最大返回数量,-1代表没有限制。",
|
||||
"Max number of hive partition values to return while list partitions, -1 means no limitation."})
|
||||
@ -1705,7 +1705,7 @@ public class Config extends ConfigBase {
|
||||
* Max thread pool size for loading external meta cache
|
||||
*/
|
||||
@ConfField(mutable = false, masterOnly = false)
|
||||
public static int max_external_cache_loader_thread_pool_size = 10;
|
||||
public static int max_external_cache_loader_thread_pool_size = 64;
|
||||
|
||||
/**
|
||||
* Max cache num of external catalog's file
|
||||
|
||||
Reference in New Issue
Block a user