database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[improve](multi-table-load) avoid plan and execute too many plan at once (#29951)

Browse Source
This commit is contained in:
HHoflittlefish777
2024-01-16 20:44:54 +08:00
committed by yiguolei
parent 8090a395bd
commit 2fa511f80e
6 changed files with 36 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
be/src/common/config.cpp
View File

@ -838,6 +838,13 @@ DEFINE_Int32(routine_load_consumer_pool_size, "10");
// if the size of batch is more than this threshold, we will request plans for all related tables.
DEFINE_Int32(multi_table_batch_plan_threshold, "200");
// Used in single-stream-multi-table load. When receiving a batch of messages from Kafka,
// if the size of the table wait for plan is more than this threshold, we will request plans for all related tables.
// The param is aimed to avoid requesting and executing too many plans at once.
// Performing small batch processing on multiple tables during the loaded process can reduce the pressure of a single RPC
// and improve the real-time processing of data.
DEFINE_Int32(multi_table_max_wait_tables, "5");
// When the timeout of a load task is less than this threshold,
// Doris treats it as a high priority task.
// high priority tasks use a separate thread pool for flush and do not block rpc by memory cleanup logic.