doris

Files

lihangyu ab8125d56f [Improve](performance) introduce SchemaCache to cache TabletSchame & Schema (#20037 )

* [Improve](performance) introduce SchemaCache to cache TabletSchame & Schema

1. When the system is under high-concurrency load with wide table point queries, the frequent memory allocation and deallocation of Schema become evident system bottlenecks. Additionally, the initialization of TabletSchema and Schema also becomes a CPU hotspot.Therefore, the introduction of a SchemaCache is implemented to cache these resources for reuse.

2. Make some variables wrapped with std::unique<unique_ptr>

Performance:
| 状态              | QPS | 平均响应时间 (avg) | P99 响应时间 |
|------------------|-----|------------------|-------------|
| 开启 SchemaCache | 501 | 20ms             | 34ms        |
| 关闭 SchemaCache | 321 | 31ms             | 61ms        |

* handle schema change with schema version

* remove useless header

* rebase

2023-05-29 17:34:53 +08:00

cache

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

memory

[fix](resource-group) Fix resource group memory isolation may release too much memory (#20066 )

2023-05-26 14:50:12 +08:00

routine_load

[Chore](build) add non-virtual-dtor, remove no-embedded-directive/no-zero-length-array (#20118 )

2023-05-29 14:42:47 +08:00

stream_load

[Feature-WIP](CCR): Add ccr doris interface (WIP) (#17881 )

2023-05-26 23:40:49 +08:00

task_group

[fix](resource-group) Fix resource group memory isolation may release too much memory (#20066 )

2023-05-26 14:50:12 +08:00

block_spill_manager.cpp

[feature](agg) Support spill to disk in aggregation (#18051 )

2023-04-20 18:59:08 +08:00

block_spill_manager.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

broker_mgr.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

broker_mgr.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

buffer_control_block.cpp

[improvement](fetch data) Merge result into batch to reduce rpc times (#17828 )

2023-04-23 15:07:28 +08:00

buffer_control_block.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

client_cache.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

client_cache.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

CMakeLists.txt

[chore](compile) using PCH for compilation acceleration under clang (#19303 )

2023-05-08 19:51:06 +08:00

collection_value.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

collection_value.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

datetime_value.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

decimalv2_value.cpp

[fix](ubsan) fix ubsan errors (#19658 )

2023-05-17 09:32:03 +08:00

decimalv2_value.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

define_primitive_type.h

[Feature](generic-aggregation) add some type define for generic aggregate functions support (#19252 )

2023-05-06 11:30:13 +08:00

descriptor_helper.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

descriptors.cpp

[WIP](row store) two phase opt read row store (#18654 )

2023-05-16 13:21:58 +08:00

descriptors.h

[WIP](row store) two phase opt read row store (#18654 )

2023-05-16 13:21:58 +08:00

exec_env_init.cpp

[Improve](performance) introduce SchemaCache to cache TabletSchame & Schema (#20037 )

2023-05-29 17:34:53 +08:00

exec_env.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

exec_env.h

[Refactor](exceptionsafe) add factory creator to some class (#19000 )

2023-04-25 14:33:47 +08:00

export_task_mgr.cpp

[Pipeline](fix) fix enable_pipeline_engine variable not work (#14909 )

2022-12-08 14:52:52 +08:00

export_task_mgr.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

external_scan_context_mgr.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

external_scan_context_mgr.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

fold_constant_executor.cpp

[refactor](exec) replace the single pointer with an array of 'conjuncts' in ExecNode (#19758 )

2023-05-29 11:47:31 +08:00

fold_constant_executor.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

fragment_mgr.cpp

[DEBUG](Log) Add debug string for pipeline task cacnel (#20026 )

2023-05-25 09:58:31 +08:00

fragment_mgr.h

[Improvement](runtime filter) Improve merge phase (#18828 )

2023-04-26 21:01:20 +08:00

heartbeat_flags.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

jsonb_value.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

jsonb_value.h

[bugfix](jsonb) fix jsonb parser crash on noavx2 host (#18977 )

2023-04-26 15:10:12 +08:00

large_int_value.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

large_int_value.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

load_channel_mgr.cpp

[Chore](build) add non-virtual-dtor, remove no-embedded-directive/no-zero-length-array (#20118 )

2023-05-29 14:42:47 +08:00

load_channel_mgr.h

[enhancement](load) lazy-open necessary partitions when load (#18874 )

2023-05-14 16:09:55 +08:00

load_channel.cpp

[enhancement](load) add some profile items for load (#20141 )

2023-05-29 09:54:03 +08:00

load_channel.h

[enhancement](load) add some profile items for load (#20141 )

2023-05-29 09:54:03 +08:00

load_path_mgr.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

load_path_mgr.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

map_value.cpp

[Feature](Complex-type) Add struct and map type to Doris (#16444 )

2023-02-10 11:00:33 +08:00

map_value.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

message_body_sink.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

message_body_sink.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

plan_fragment_executor.cpp

[improvement](profile) add profile counter 'BytesSent' for VDataBufferSender (#19826 )

2023-05-19 08:46:50 +08:00

plan_fragment_executor.h

[improvement](tracing) Remove useless span levels from be side tracing (#19665 )

2023-05-17 19:04:52 +08:00

primitive_type.cpp

[Feature](generic-aggregation) add some type define for generic aggregate functions support (#19252 )

2023-05-06 11:30:13 +08:00

primitive_type.h

[Feature](generic-aggregation) some prowork of generic aggregation (#19343 )

2023-05-09 21:42:21 +08:00

query_context.h

[fix](resource-group) Fix resource group memory isolation may release too much memory (#20066 )

2023-05-26 14:50:12 +08:00

query_statistics.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

query_statistics.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

raw_value.h

[unify type system](remove unused type desc) remove some code (#17921 )

2023-03-19 14:05:02 +08:00

record_batch_queue.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

record_batch_queue.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

result_buffer_mgr.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

result_buffer_mgr.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

result_queue_mgr.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

result_queue_mgr.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

result_writer.cpp

[OUTFILE] Support INTO OUTFILE to export query result (#3584 )

2020-05-25 21:24:56 +08:00

result_writer.h

[Chore](build) add non-virtual-dtor, remove no-embedded-directive/no-zero-length-array (#20118 )

2023-05-29 14:42:47 +08:00

runtime_filter_mgr.cpp

[Improvement](runtime filter) Improve merge phase (#18828 )

2023-04-26 21:01:20 +08:00

runtime_filter_mgr.h

[Improvement](runtime filter) Improve merge phase (#18828 )

2023-04-26 21:01:20 +08:00

runtime_predicate.cpp

[bugfix](topn) fix memory leak in topn AcceptNullPredicate (#19060 )

2023-04-27 14:07:57 +08:00

runtime_predicate.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

runtime_state.cpp

[fix](config) fix memory config enable_query_memroy_overcommit spell problem #19898

2023-05-22 00:32:20 +08:00

runtime_state.h

[improvement](status) should send query timeout status to be, instead of internal error (#20016 )

2023-05-26 15:11:17 +08:00

small_file_mgr.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

small_file_mgr.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

snapshot_loader.cpp

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

snapshot_loader.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

string_search.hpp

[refactor](string) remove volnitsky search algorithm (#18474 )

2023-04-10 10:56:07 +08:00

struct_value.cpp

[Feature](Complex-type) Add struct and map type to Doris (#16444 )

2023-02-10 11:00:33 +08:00

struct_value.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

tablets_channel.cpp

[enhancement](load) add some profile items for load (#20141 )

2023-05-29 09:54:03 +08:00

tablets_channel.h

[enhancement](load) add some profile items for load (#20141 )

2023-05-29 09:54:03 +08:00

thread_context.cpp

[fix](memory) query check cancel while waiting for memory in Allocator, and optimize log (#19967 )

2023-05-24 11:08:48 +08:00

thread_context.h

[fix](memory) query check cancel while waiting for memory in Allocator, and optimize log (#19967 )

2023-05-24 11:08:48 +08:00

threadlocal.cc

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

threadlocal.h

[chore](build) Use include-what-you-use to optimize includes (PART II) (#18761 )

2023-04-19 23:11:48 +08:00

type_limit.h

[Bug](runtime filter) Fix min/max filter for decimalv3 (#20005 )

2023-05-26 21:35:21 +08:00

types.cpp

[FIX](Array)fix be master comapitible with fe1.2 #19850

2023-05-19 14:10:29 +08:00

types.h

[Feature](agg_state) support agg_state combinators (#19969 )

2023-05-29 13:07:29 +08:00

user_function_cache.cpp

[bugfix](memleak) UserFunctionCache may have memory leak during close (#18913 )

2023-04-22 10:15:51 +08:00

user_function_cache.h

[bugfix](memleak) UserFunctionCache may have memory leak during close (#18913 )

2023-04-22 10:15:51 +08:00