database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[Improvement](inverted index) inverted index query match bitmap cache (#16578)

Browse Source 
Add cache for inverted index query match bitmap to accelerate common query keyword, especially for keyword matching many rows. 

Tests result:
- large result: matching 99% out of 247 million rows shows 8x speed up.
- small result: matching 0.1% out of 247 million rows shows 2x speed up.
This commit is contained in:
Kang
2023-02-11 13:38:58 +08:00
committed by GitHub
parent 37d1519316
commit aba843bb2b
16 changed files with 419 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
be/src/runtime/exec_env_init.cpp
View File

@ -233,6 +233,19 @@ Status ExecEnv::_init_mem_env() {
<< PrettyPrinter::print(inverted_index_cache_limit, TUnit::BYTES)
<< ", origin config value: " << config::inverted_index_searcher_cache_limit;
// use memory limit
int64_t inverted_index_query_cache_limit =
ParseUtil::parse_mem_spec(config::inverted_index_query_cache_limit,
MemInfo::mem_limit(), MemInfo::physical_mem(), &is_percent);
while (!is_percent && inverted_index_query_cache_limit > MemInfo::mem_limit() / 2) {
// Reason same as buffer_pool_limit
inverted_index_query_cache_limit = inverted_index_query_cache_limit / 2;
}
InvertedIndexQueryCache::create_global_cache(inverted_index_query_cache_limit, 10);
LOG(INFO) << "Inverted index query match cache memory limit: "
<< PrettyPrinter::print(inverted_index_cache_limit, TUnit::BYTES)
<< ", origin config value: " << config::inverted_index_query_cache_limit;
// 4. init other managers
RETURN_IF_ERROR(_tmp_file_mgr->init());
RETURN_IF_ERROR(_block_spill_mgr->init());