hui lai d4c1b39d03 [fix](multi table) restrict the multi tables load memory under high concurrency with a large number of tables (#39992) (#41131) ...

pick (#39992)

BE node was killed by OOM-killer when use multi table load under high
concurrency with a large number of tables(128 concurrency and every
concurrency load 200 tables).

This pr restricts the multi tables load memory under this issue. If
memory reaches hard limit, new task will be rejected and return
directly.

2024-09-24 16:34:32 +08:00

..

cache

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

memory

[branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 (#40850)

2024-09-15 23:47:53 +08:00

routine_load

[fix](multi table) restrict the multi tables load memory under high concurrency with a large number of tables (#39992) (#41131)

2024-09-24 16:34:32 +08:00

stream_load

[improvement](stream load) display the time of receiving network data (#40735) (#41139)

2024-09-24 16:00:28 +08:00

workload_group

[fix](systable) fix unstable case for partitions table (#40553) (#41043)

2024-09-20 17:13:30 +08:00

workload_management

[branch-2.1]Reset io limit default value (#39898)

2024-08-26 14:27:35 +08:00

block_spill_manager.cpp

[enhance](FS) Improve FS error code (#29432)

2024-01-06 21:17:22 +08:00

block_spill_manager.h

…

broker_mgr.cpp

[Refactor & Bugfix](static variables) move some static vairables to exec_env (#24029)

2023-09-13 09:27:03 +08:00

broker_mgr.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

buffer_control_block.cpp

[fix](execution) fix wrong check when blocking result sink (#40289)

2024-09-03 14:17:42 +08:00

buffer_control_block.h

[fix](sink) Do not block result sink on pipeline engine (#40094)

2024-08-29 14:04:25 +08:00

client_cache.cpp

[opt](fqdn) Add DNS Cache for FE and BE (#32869)

2024-04-07 22:16:04 +08:00

client_cache.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

CMakeLists.txt

Revert "[fix](compile) fix two compile errors on MacOS (#33834) (#34005)"

2024-04-26 00:55:21 +08:00

collection_value.cpp

…

collection_value.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

decimalv2_value.cpp

[Chore](build) add -Wconversion and remove some unused code (#33127)

2024-04-10 15:26:08 +08:00

decimalv2_value.h

[refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits (#26191)

2023-11-15 21:34:49 +08:00

define_primitive_type.h

[Feature](datatype) Add IPv4/v6 data type for doris (#24965)

2023-10-26 17:33:28 +08:00

descriptor_helper.h

[Chore](descriptors) remove unused codes for descriptors (#33408)

2024-04-12 15:09:25 +08:00

descriptors.cpp

[fix](Nereids) fix Illegal aggregate node: group by and output is empty (#35497)

2024-05-29 15:01:47 +08:00

descriptors.h

[fix](ub) fix uninitialized accesses in BE (#35370)

2024-05-29 20:31:07 +08:00

exec_env_init.cpp

[fix](multi table) restrict the multi tables load memory under high concurrency with a large number of tables (#39992) (#41131)

2024-09-24 16:34:32 +08:00

exec_env.cpp

[feature](api) add BE HTTP /api/load_streams (#36312) (#36338)

2024-06-16 22:09:04 +08:00

exec_env.h

[branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 (#40850)

2024-09-15 23:47:53 +08:00

external_scan_context_mgr.cpp

[Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly (#23395)

2023-09-29 22:38:52 +08:00

external_scan_context_mgr.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

fold_constant_executor.cpp

[branch-2.1](memory) Allocator support address sanitizers (#40836)

2024-09-14 12:12:44 +08:00

fold_constant_executor.h

[opt](memory) Optimize mem tracker accuracy (#32039) (#33140)

2024-04-10 11:42:19 +08:00

fragment_mgr.cpp

[fix](pipelinex) fix fragment instance progress reports (#40325) (#40987)

2024-09-19 23:58:38 +08:00

fragment_mgr.h

[opt](query cancel) cancel query if it has pipeline task leakage #39223 (#39537)

2024-08-19 14:33:59 +08:00

frontend_info.h

[opt](query cancel) cancel query if it has pipeline task leakage #39223 (#39537)

2024-08-19 14:33:59 +08:00

group_commit_mgr.cpp

[opt](query cancel) cancel query if it has pipeline task leakage #39223 (#39537)

2024-08-19 14:33:59 +08:00

group_commit_mgr.h

[fix](group commit) Pick some group commit pr (#38320)

2024-07-25 17:32:44 +08:00

heartbeat_flags.h

…

jsonb_value.cpp

[Fix](json) avoid print warn log when parse failed (#30656)

2024-02-01 19:00:50 +08:00

jsonb_value.h

[Fix](compile) Fix gcc compile on master (#33864)

2024-04-19 23:41:37 +08:00

large_int_value.cpp

…

large_int_value.h

[improve](fmt) use format_to and FMT_COMPILE to speed up write data #29682

2024-01-12 11:53:57 +08:00

load_channel_mgr.cpp

[branch-2.1](auto-partition) Fix auto partition load failure in multi replica (#36586)

2024-06-20 17:51:18 +08:00

load_channel_mgr.h

[branch-2.1](memory) Refactor LRU cache policy memory tracking (#37658)

2024-07-11 21:04:01 +08:00

load_channel.cpp

[branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth (#39760)

2024-08-22 17:33:11 +08:00

load_channel.h

[branch-2.1](auto-partition) Fix auto partition load failure in multi replica (#36586)

2024-06-20 17:51:18 +08:00

load_path_mgr.cpp

[log](load) print reason when LoadPathMgr failed to delete file (#38235) (#38340)

2024-07-25 17:28:09 +08:00

load_path_mgr.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

load_stream_mgr.cpp

[feature](api) add BE HTTP /api/load_streams (#36312) (#36338)

2024-06-16 22:09:04 +08:00

load_stream_mgr.h

[feature](api) add BE HTTP /api/load_streams (#36312) (#36338)

2024-06-16 22:09:04 +08:00

load_stream_writer.cpp

[fix](move-memtable) fix bvar g_load_stream_file_writer_cnt (#39075) (#40985)

2024-09-20 20:20:49 +08:00

load_stream_writer.h

[opt](memory) Optimize mem tracker accuracy (#32039) (#33140)

2024-04-10 11:42:19 +08:00

load_stream.cpp

[fix](move-memtable) do not execute close if create rowset failed when loading MOW table (#40105) (#41132)

2024-09-24 16:02:22 +08:00

load_stream.h

[fix](move-memtable) multi replica tables should tolerate minority failures (#38003) (#40477)

2024-09-09 11:30:46 +08:00

map_value.cpp

…

map_value.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

message_body_sink.cpp

…

message_body_sink.h

…

plan_fragment_executor.cpp

[opt](memory) Optimize mem tracker accuracy (#32039) (#33140)

2024-04-10 11:42:19 +08:00

plan_fragment_executor.h

[opt](memory) Optimize mem tracker accuracy (#32039) (#33140)

2024-04-10 11:42:19 +08:00

primitive_type.cpp

[Feature](datatype) Add IPv4/v6 data type for doris (#24965)

2023-10-26 17:33:28 +08:00

primitive_type.h

[Fix](compile) Fix gcc compile on master (#33864)

2024-04-19 23:41:37 +08:00

query_context.cpp

[opt](query cancel) cancel query if it has pipeline task leakage #39223 (#39537)

2024-08-19 14:33:59 +08:00

query_context.h

[branch-2.1](memory) pick reserve memory and workload group (#40543)

2024-09-09 21:16:06 +08:00

query_statistics.cpp

[refactor](raw ptr) disable some raw pointer usage and some unused code (#31595)

2024-02-29 19:51:18 +08:00

query_statistics.h

[refactor](raw ptr) disable some raw pointer usage and some unused code (#31595)

2024-02-29 19:51:18 +08:00

raw_value.h

[feature](decimal) support decimal256 (#25386)

2023-10-25 15:47:51 +08:00

record_batch_queue.cpp

…

record_batch_queue.h

[pipeline](sink) Supprt Async Writer Sink of result file sink and memory scratch sink (#23589)

2023-08-31 22:44:25 +08:00

result_buffer_mgr.cpp

[Fix](Status) Handle returned overall Status correctly (#31692)

2024-03-09 19:44:39 +08:00

result_buffer_mgr.h

[Fix](Status) Handle returned overall Status correctly (#31692)

2024-03-09 19:44:39 +08:00

result_queue_mgr.cpp

…

result_queue_mgr.h

…

result_writer.h

[refactor](close) refactor ispendingfinish logic and close logic to do close more quickly (#30021)

2024-01-23 10:06:05 +08:00

runtime_filter_mgr.cpp

[fix](brpc_client_cache) resolve hostname in DNS cache before passing to brpc (#40074) (#40786)

2024-09-13 14:28:01 +08:00

runtime_filter_mgr.h

[Improvement](runtime-filter) support sync join node build side's size to init bloom runtime filter (#32180)

2024-04-11 09:31:50 +08:00

runtime_predicate.cpp

[fix](type)support runtime predicate for time type (#38258) (#38465)

2024-07-31 10:27:36 +08:00

runtime_predicate.h

[Bug](top-n) do not get runtime predicate when predicate not initialized (#32208)

2024-03-15 18:06:15 +08:00

runtime_query_statistics_mgr.cpp

[fix](systable) fix unstable case for partitions table (#40553) (#41043)

2024-09-20 17:13:30 +08:00

runtime_query_statistics_mgr.h

[opt](file-cache) support system table file_cache_statistics (#39552)

2024-08-21 10:03:39 +08:00

runtime_state.cpp

[cherrypick](branch21) pick some exception handleing pr in page io (#40883)

2024-09-16 08:56:43 +08:00

runtime_state.h

[cherrypick](branch21) pick some exception handleing pr in page io (#40883)

2024-09-16 08:56:43 +08:00

small_file_mgr.cpp

…

small_file_mgr.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

snapshot_loader.cpp

[chore](fe) Returns dropped tables in GetMeta request (#38541)

2024-07-31 10:57:00 +08:00

snapshot_loader.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

string_search.hpp

…

struct_value.cpp

…

struct_value.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

tablets_channel.cpp

[branch-2.1](load) Lower lock granularity to reduce loading CPU pressure (#40135)

2024-08-29 23:22:58 +08:00

tablets_channel.h

[branch-2.1](load) fix miss writer in concurrency incremental open (#38605) (#38793)

2024-08-05 08:56:23 +08:00

task_execution_context.h

[memoryleak](execontext) dctor should be virtual to avoid memory leak (#30359)

2024-01-25 21:37:33 +08:00

thread_context.cpp

[branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth (#39760)

2024-08-22 17:33:11 +08:00

thread_context.h

[branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 (#40850)

2024-09-15 23:47:53 +08:00

type_limit.h

[refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion (#29265)

2023-12-29 10:11:44 +08:00

types.cpp

[exec](runtimefilter) support null aware in runtime filter (#32152)

2024-03-15 18:05:13 +08:00

types.h

[feature](decimal) support decimal256 (#25386)

2023-10-25 15:47:51 +08:00

user_function_cache.cpp

[improvement](spinlock) remove some potential bad spinlock usage (#27904)

2023-12-02 20:33:54 +08:00

user_function_cache.h

[fix](jdbc catalog) fix jdbc driver cache load error (#23656)

2023-08-31 10:17:15 +08:00