[Enhancement](multi-catalog) Make meta cache batch loading concurrently. (#21471)

I will enhance performance about querying meta cache of hms tables by 2 steps:
**Step1** : use concurrent batch loading for meta cache
**Step2** : execute some other tasks concurrently as soon as possible

**This pr mainly for step1 and it mainly do the following things:**
- Create a `CacheBulkLoader` for batch loading
- Remove the executor of the previous async cache loader and change the loader's type to `CacheBulkLoader` (We do not set any refresh strategies for LoadingCache, so the previous executor is not useful)
- Use a `FixedCacheThreadPool` to replace the `CacheThreadPool` (The previous `CacheThreadPool` just log warn infos and will not throw any exceptions when the pool is full).
- Remove parallel streams and use the `CacheBulkLoader` to do batch loadings
- Change the value of `max_external_cache_loader_thread_pool_size` to 64, and set the pool size of hms client pool to `max_external_cache_loader_thread_pool_size`
- Fix the spelling mistake for `max_hive_table_catch_num`
This commit is contained in:
Xiangyu Wang
2023-07-06 15:18:30 +08:00
committed by GitHub
parent fde73b6cc6
commit bb3b6770b5
8 changed files with 233 additions and 101 deletions

View File

@ -1694,7 +1694,7 @@ public class Config extends ConfigBase {
@ConfField(mutable = false, masterOnly = false, description = {"Hive表到分区名列表缓存的最大数量。",
"Max cache number of hive table to partition names list."})
public static long max_hive_table_catch_num = 1000;
public static long max_hive_table_cache_num = 1000;
@ConfField(mutable = false, masterOnly = false, description = {"获取Hive分区值时候的最大返回数量,-1代表没有限制。",
"Max number of hive partition values to return while list partitions, -1 means no limitation."})
@ -1705,7 +1705,7 @@ public class Config extends ConfigBase {
* Max thread pool size for loading external meta cache
*/
@ConfField(mutable = false, masterOnly = false)
public static int max_external_cache_loader_thread_pool_size = 10;
public static int max_external_cache_loader_thread_pool_size = 64;
/**
* Max cache num of external catalog's file