[enhancement](execute model) using thread pool to execute report or join task instead of staring too many thread (#17212)
* [enhancement](execute model) using thread pool to execute report or join task instead of staring too many thread Doris will start report thread and join thread during fragment execution. There are many problems if create and destroy thread very frequently. Jemalloc may not behave very well, it may crashed. jemalloc/jemalloc#1405 It is better to using thread pool to do these tasks. --------- Co-authored-by: yiguolei <yiguolei@gmail.com>
This commit is contained in:
@ -102,6 +102,21 @@ Status ExecEnv::_init(const std::vector<StorePath>& store_paths) {
|
||||
|
||||
init_download_cache_required_components();
|
||||
|
||||
// min num equal to fragment pool's min num
|
||||
// max num is useless because it will start as many as requested in the past
|
||||
// queue size is useless because the max thread num is very large
|
||||
ThreadPoolBuilder("SendReportThreadPool")
|
||||
.set_min_threads(config::fragment_pool_thread_num_min)
|
||||
.set_max_threads(std::numeric_limits<int>::max())
|
||||
.set_max_queue_size(config::fragment_pool_queue_size)
|
||||
.build(&_send_report_thread_pool);
|
||||
|
||||
ThreadPoolBuilder("JoinNodeThreadPool")
|
||||
.set_min_threads(config::fragment_pool_thread_num_min)
|
||||
.set_max_threads(std::numeric_limits<int>::max())
|
||||
.set_max_queue_size(config::fragment_pool_queue_size)
|
||||
.build(&_join_node_thread_pool);
|
||||
|
||||
RETURN_IF_ERROR(init_pipeline_task_scheduler());
|
||||
_scanner_scheduler = new doris::vectorized::ScannerScheduler();
|
||||
|
||||
|
||||
Reference in New Issue
Block a user