database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[fix](multi table) restrict the multi tables load memory under high concurrency with a large number of tables (#39992) (#41131)

Browse Source 
pick (#39992)

BE node was killed by OOM-killer when use multi table load under high
concurrency with a large number of tables(128 concurrency and every
concurrency load 200 tables).

This pr restricts the multi tables load memory under this issue. If
memory reaches hard limit, new task will be rejected and return
directly.
This commit is contained in:
hui lai
2024-09-24 16:34:32 +08:00
committed by GitHub
parent 3ad9dce067
commit d4c1b39d03
4 changed files with 23 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
be/src/runtime/exec_env_init.cpp
View File

@ -239,7 +239,7 @@ Status ExecEnv::_init(const std::vector<StorePath>& store_paths,
new BrpcClientCache<PFunctionService_Stub>(config::function_service_protocol);
_stream_load_executor = StreamLoadExecutor::create_shared(this);
_routine_load_task_executor = new RoutineLoadTaskExecutor(this);
RETURN_IF_ERROR(_routine_load_task_executor->init());
RETURN_IF_ERROR(_routine_load_task_executor->init(MemInfo::mem_limit()));
_small_file_mgr = new SmallFileMgr(this, config::small_file_dir);
_block_spill_mgr = new BlockSpillManager(store_paths);
_group_commit_mgr = new GroupCommitMgr(this);
@ -540,7 +540,7 @@ void ExecEnv::init_mem_tracker() {
_s3_file_buffer_tracker =
MemTrackerLimiter::create_shared(MemTrackerLimiter::Type::GLOBAL, "S3FileBuffer");
_stream_load_pipe_tracker =
MemTrackerLimiter::create_shared(MemTrackerLimiter::Type::GLOBAL, "StreamLoadPipe");
MemTrackerLimiter::create_shared(MemTrackerLimiter::Type::LOAD, "StreamLoadPipe");
}
void ExecEnv::_register_metrics() {