Sun Chenyang 2e1318b8a0 [fix] (compaction) fix CompactionPermitLimiter causing compaction to stall (#35078) ...

## BUG
1. config::total_permits_for_compaction_score = 20000
2. Thread-B requests permits 11000， used_permits = 11000
3. Thread-A requests permits 12000，wait for used_permits + 12000 <=
20000
4.  adjust config::total_permits_for_compaction_score = 10000
5. Thread-B releases permits，used_permits = 0，notify
Thread-A，used_permits + 12000 <= 10000

## FIx
we need to initialize total_permits instead of using the config

2024-05-28 18:52:34 +08:00

..

rowset

[opt](scan) Use lazy-init for segment iterators and avoid caching all segments in the rowset reader (#35432)

2024-05-28 13:19:18 +08:00

task

[Fix](inverted index) fix race condition in index build (#35427)

2024-05-28 13:12:03 +08:00

wal

[refactor](datalake) return the error status instead of static_cast<void> (#34873)

2024-05-23 19:06:21 +08:00

accept_null_predicate.h

[Bug](regression) fix bug of null in topn opt (#31669)

2024-03-02 17:12:34 +08:00

base_compaction.cpp

Prevent making snapshot on remote rowset in single replica compaction (#28716)

2023-12-27 23:43:43 +08:00

base_compaction.h

Prevent making snapshot on remote rowset in single replica compaction (#28716)

2023-12-27 23:43:43 +08:00

base_tablet.cpp

[improve] Use lru cache to count the number of column in tablet schema to control memory (#29668)

2024-01-12 13:58:19 +08:00

base_tablet.h

[Improvement](variant) limit the column size on tablet schema (#27399) (#27785)

2023-12-04 14:47:36 +08:00

binlog_config.cpp

…

binlog_config.h

…

binlog.h

…

bitmap_filter_predicate.h

[Feature](runtime filter) normalize ignore runtime filter (#30152)

2024-02-03 20:24:39 +08:00

block_column_predicate.cpp

[refactor](exception safe) Refactor delete handler and block column predicates to make sure exception safe (#31618)

2024-03-01 14:21:17 +08:00

block_column_predicate.h

[refactor](exception safe) Refactor delete handler and block column predicates to make sure exception safe (#31618)

2024-03-01 14:21:17 +08:00

bloom_filter_predicate.h

[Feature](runtime filter) normalize ignore runtime filter (#30152)

2024-02-03 20:24:39 +08:00

bloom_filter.hpp

…

calc_delete_bitmap_executor.cpp

[opt](memory) Optimize mem tracker accuracy (#32039) (#33140)

2024-04-10 11:42:19 +08:00

calc_delete_bitmap_executor.h

[opt](memory) Optimize mem tracker accuracy (#32039) (#33140)

2024-04-10 11:42:19 +08:00

CMakeLists.txt

…

cold_data_compaction.cpp

[feature](merge-cloud) Decouple Tablet/TabletManager/TxnManager from global StorageEngine instance (#29736)

2024-01-12 11:57:16 +08:00

cold_data_compaction.h

…

column_mapping.h

[Bug](schame-change) fix wrong result after reorder mor table (#29045)

2023-12-28 14:57:31 +08:00

column_predicate.h

[opt](inverted index) add inlist condition handling to compound (#34134)

2024-05-10 14:35:47 +08:00

compaction_permit_limiter.cpp

[fix] (compaction) fix CompactionPermitLimiter causing compaction to stall (#35078)

2024-05-28 18:52:34 +08:00

compaction_permit_limiter.h

…

compaction.cpp

[fix](inverted index) the rowset may be deleted and become nullptr. #33878 (#34464)

2024-05-08 09:11:22 +08:00

compaction.h

[fix](path-gc) Fix pending rowset guard check failure when ordered data compaction failed (#33029)

2024-03-29 17:47:51 +08:00

comparison_predicate.h

[Fix](compile) Fix gcc compile on master (#33864)

2024-04-19 23:41:37 +08:00

cumulative_compaction_policy.cpp

[Cherry-pick](branch-2.1) Pick #34043 and #34112 (#34318)

2024-04-30 07:26:23 +08:00

cumulative_compaction_policy.h

(invert index) modify of time series compaction policy

2024-03-22 08:16:30 +08:00

cumulative_compaction_time_series_policy.cpp

[fix](compaction) fix time series compaction merge empty rowsets priority #34562 (#34765)

2024-05-14 09:10:09 +08:00

cumulative_compaction_time_series_policy.h

(invert index) modify of time series compaction policy

2024-03-22 08:16:30 +08:00

cumulative_compaction.cpp

(invert index) modify of time series compaction policy

2024-03-22 08:16:30 +08:00

cumulative_compaction.h

Prevent making snapshot on remote rowset in single replica compaction (#28716)

2023-12-27 23:43:43 +08:00

data_dir.cpp

[improve](disk) Not add disk path to broken list if check status is not IO_ERROR (#34111)

2024-04-26 07:44:12 +08:00

data_dir.h

[feature](spill) Implement spill to disk for hash join, aggregation and sort for pipelineX (#31910)

2024-03-12 14:12:09 +08:00

decimal12.h

[refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits (#26191)

2023-11-15 21:34:49 +08:00

delete_bitmap_calculator.cpp

[feature-wip](merge-on-write) MOW table support different primary keys and sort keys (#24788)

2023-11-24 16:37:30 +08:00

delete_bitmap_calculator.h

[feature-wip](merge-on-write) MOW table support different primary keys and sort keys (#24788)

2023-11-24 16:37:30 +08:00

delete_handler.cpp

[refactor](exception safe) Refactor delete handler and block column predicates to make sure exception safe (#31618)

2024-03-01 14:21:17 +08:00

delete_handler.h

[refactor](exception safe) Refactor delete handler and block column predicates to make sure exception safe (#31618)

2024-03-01 14:21:17 +08:00

delta_writer_context.h

[fix](move-memtable) fix schema use-after-free in delta writer v2 (#30254)

2024-01-24 10:00:25 +08:00

delta_writer_v2.cpp

[Cherry-Pick](branch-2.1) Pick "Fix multiple replica partial update auto inc data inconsistency problem #34788" (#35056)

2024-05-20 15:43:46 +08:00

delta_writer_v2.h

[Refactor](Sink) Remove is_append mode in table sink (#34684)

2024-05-11 11:20:10 +08:00

delta_writer.cpp

[fix](compaction) fix time series compaction merge empty rowsets priority #34562 (#34765)

2024-05-14 09:10:09 +08:00

delta_writer.h

[Refactor](Sink) Remove is_append mode in table sink (#34684)

2024-05-11 11:20:10 +08:00

field.h

[Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path (#31141)

2024-03-09 19:44:42 +08:00

file_header.h

[enhance](FS) Improve FS error code (#29432)

2024-01-06 21:17:22 +08:00

full_compaction.cpp

[Cherry-pick](branch-2.1) Pick #34043 and #34112 (#34318)

2024-04-30 07:26:23 +08:00

full_compaction.h

Prevent making snapshot on remote rowset in single replica compaction (#28716)

2023-12-27 23:43:43 +08:00

hll.cpp

…

hll.h

[exec](column) change some complex column move to noexcept (#32954)

2024-04-10 11:34:29 +08:00

in_list_predicate.h

[Fix](compile) Fix gcc compile on master (#33864)

2024-04-19 23:41:37 +08:00

inverted_index_parser.cpp

[opt](inverted index) the "unicode" tokenizer can be configured to disable stop words. (#34467)

2024-05-07 18:23:43 +08:00

inverted_index_parser.h

[opt](inverted index) the "unicode" tokenizer can be configured to disable stop words. (#34467)

2024-05-07 18:23:43 +08:00

iterators.h

[fix](auto inc) db_id and table_id should be int64_t instead of int32_t (#34912)

2024-05-18 18:29:59 +08:00

itoken_extractor.cpp

…

itoken_extractor.h

…

key_coder.cpp

[Feature](datatype) Add IPv4/v6 data type for doris (#24965)

2023-10-26 17:33:28 +08:00

key_coder.h

[feature](decimal) support decimal256 (#25386)

2023-10-25 15:47:51 +08:00

like_column_predicate.cpp

[Feature](runtime filter) normalize ignore runtime filter (#30152)

2024-02-03 20:24:39 +08:00

like_column_predicate.h

[Fix](compile) Fix gcc compile on master (#33864)

2024-04-19 23:41:37 +08:00

lru_cache.cpp

[fix](memory) Fix LRU cache deleter and memory tracking (#32080)

2024-03-15 17:57:58 +08:00

lru_cache.h

[fix](memory) Fix LRU cache deleter and memory tracking (#32080)

2024-03-15 17:57:58 +08:00

match_predicate.cpp

[Optimize](inverted index) optimize inverted index bitmap copy (#32279) (#32469)

2024-03-19 17:28:59 +08:00

match_predicate.h

[Fix](compile) Fix gcc compile on master (#33864)

2024-04-19 23:41:37 +08:00

memtable_flush_executor.cpp

[enhancement](memory) return error if allocate memory failed during add rows method (#35085)

2024-05-22 00:53:34 +08:00

memtable_flush_executor.h

[Feature](executor)Insert select limited by WorkloadGroup #30610

2024-01-31 23:53:40 +08:00

memtable_memory_limiter.cpp

[opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory (#34985) (#35070)

2024-05-22 09:53:45 +08:00

memtable_memory_limiter.h

[opt](memory) Optimize mem tracker accuracy (#32039) (#33140)

2024-04-10 11:42:19 +08:00

memtable_writer.cpp

[enhancement](memory) return error if allocate memory failed during add rows method (#35085)

2024-05-22 00:53:34 +08:00

memtable_writer.h

[Refactor](Sink) Remove is_append mode in table sink (#34684)

2024-05-11 11:20:10 +08:00

memtable.cpp

[enhancement](memory) return error if allocate memory failed during add rows method (#35085)

2024-05-22 00:53:34 +08:00

memtable.h

[enhancement](memory) return error if allocate memory failed during add rows method (#35085)

2024-05-22 00:53:34 +08:00

merger.cpp

(pick 34003) release fd for shutdown tablets (#34224)

2024-04-29 10:51:19 +08:00

merger.h

[feature-wip](merge-on-write) MOW table support different primary keys and sort keys (#24788)

2023-11-24 16:37:30 +08:00

null_predicate.cpp

[Feature](runtime filter) normalize ignore runtime filter (#30152)

2024-02-03 20:24:39 +08:00

null_predicate.h

[Feature](runtime filter) normalize ignore runtime filter (#30152)

2024-02-03 20:24:39 +08:00

olap_common.h

[Cherry-pick](branch-2.1) Pick #34043 and #34112 (#34318)

2024-04-30 07:26:23 +08:00

olap_define.h

[feature](spill) Implement spill to disk for hash join, aggregation and sort for pipelineX (#31910)

2024-03-12 14:12:09 +08:00

olap_meta.cpp

[chore] Add bvar for meta operations of BE (#28374)

2023-12-19 15:54:19 +08:00

olap_meta.h

…

olap_server.cpp

[fix](compaction) fix repeatedly picking tablets with disable auto compaction (#35472) (#35505)

2024-05-28 15:57:54 +08:00

olap_tuple.h

…

options.cpp

[opt](file cache) make the cache queue percentage configurable (#29537)

2024-01-12 11:36:58 +08:00

options.h

[opt](file cache) make the cache queue percentage configurable (#29537)

2024-01-12 11:36:58 +08:00

page_cache.cpp

[opt](memory) Optimize mem tracker accuracy (#32039) (#33140)

2024-04-10 11:42:19 +08:00

page_cache.h

[fix](memory) Fix page cache memory tracker consumption in prune (#34320)

2024-05-06 12:53:11 +08:00

parallel_scanner_builder.cpp

[fix](scan) Fix missing shared tablet header lock (#31433)

2024-02-26 23:39:10 +08:00

parallel_scanner_builder.h

[feature](scan) Implement parallel scanning by dividing the tablets based on the row range (#28967)

2023-12-26 17:18:41 +08:00

partial_update_info.h

[Cherry-Pick](branch-2.1) Pick "Fix multiple replica partial update auto inc data inconsistency problem #34788" (#35056)

2024-05-20 15:43:46 +08:00

pb_helper.h

…

predicate_creator.h

[refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits (#26191)

2023-11-15 21:34:49 +08:00

primary_key_index.cpp

[enhancement](mow) Add bvar for bloom filter and segment (#32355)

2024-03-22 08:52:12 +08:00

primary_key_index.h

[enhancement](mow) Add bvar for bloom filter and segment (#32355)

2024-03-22 08:52:12 +08:00

push_handler.cpp

[opt](memory) Optimize mem tracker accuracy (#32039) (#33140)

2024-04-10 11:42:19 +08:00

push_handler.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

row_cursor_cell.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

row_cursor.cpp

[Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path (#31141)

2024-03-09 19:44:42 +08:00

row_cursor.h

[Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path (#31141)

2024-03-09 19:44:42 +08:00

rowid_conversion.h

…

rowset_builder.cpp

[opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory (#34985) (#35070)

2024-05-22 09:53:45 +08:00

rowset_builder.h

[feature](merge-cloud) Refactor write path code by abstract base class (#26537)

2023-12-08 14:50:36 +08:00

schema_cache.cpp

[fix](auto inc) db_id and table_id should be int64_t instead of int32_t (#34912)

2024-05-18 18:29:59 +08:00

schema_cache.h

[fix](auto inc) db_id and table_id should be int64_t instead of int32_t (#34912)

2024-05-18 18:29:59 +08:00

schema_change.cpp

[branch-2.1](regression-test) fix test_chema_change_fail (#33753) #33788

2024-04-17 23:42:13 +08:00

schema_change.h

[fix](schema change) fix schema change check does not calculate reader merged rows (#33825) (#33908)

2024-04-19 22:57:25 +08:00

schema.cpp

[Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path (#31141)

2024-03-09 19:44:42 +08:00

schema.h

[Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path (#31141)

2024-03-09 19:44:42 +08:00

segment_loader.cpp

[Fix](bloom filter) Fix bloom filter memory leak (#34871)

2024-05-24 16:23:58 +08:00

segment_loader.h

[Fix](bloom filter) Fix bloom filter memory leak (#34871)

2024-05-24 16:23:58 +08:00

selection_vector.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

shared_predicate.h

[Opt](Exec) Support runtime update topn filter (#31250)

2024-02-29 12:38:03 +08:00

short_key_index.cpp

…

short_key_index.h

[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default (#27587)

2023-11-28 13:02:30 +08:00

single_replica_compaction.cpp

[fix](compaction test) show single replica compaction status and fix test (#33076) (#34285) (#34438)

2024-05-06 21:00:34 +08:00

single_replica_compaction.h

[fix](compaction) fix single compaction (#33907)

2024-04-19 23:30:25 +08:00

snapshot_manager.cpp

[improvement](binlog)Support inverted index format v2 in CCR (#33415)

2024-04-17 23:42:12 +08:00

snapshot_manager.h

[branch-2.1](memory) Fix EngineStorageMigrationTask MemTracker attach thread (#33509)

2024-04-11 08:35:13 +08:00

storage_engine.cpp

[fix] fix nullptr when clear cache due to move (#34323)

2024-04-30 09:52:41 +08:00

storage_engine.h

[improve](path gc) Execute path gc interval adaptive to disk size (#34538)

2024-05-11 15:04:35 +08:00

storage_policy.cpp

[chore](StoragePolicy) Don't print stack when invoking the storage manager #28091

2023-12-07 10:06:31 +08:00

storage_policy.h

…

tablet_fwd.h

[feature](merge-cloud) Abstract BaseTablet for CloudTablet (#24929)

2023-10-18 20:29:04 +08:00

tablet_manager.cpp

[fix](compaction) fix repeatedly picking tablets with disable auto compaction (#35472) (#35505)

2024-05-28 15:57:54 +08:00

tablet_manager.h

[fix](partition) Fix be tablet partition id eq 0 By report tablet (#32179) (#32667)

2024-03-22 15:38:58 +08:00

tablet_meta_manager.cpp

[fix](typo) typo fix for 'delete bimap' changing to 'delete bitmap' (#32341)

2024-04-10 11:34:30 +08:00

tablet_meta_manager.h

[fix](typo) typo fix for 'delete bimap' changing to 'delete bitmap' (#32341)

2024-04-10 11:34:30 +08:00

tablet_meta.cpp

[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)

2024-04-10 14:53:56 +08:00

tablet_meta.h

[fix](typo) typo fix for 'delete bimap' changing to 'delete bitmap' (#32341)

2024-04-10 11:34:30 +08:00

tablet_reader.cpp

[opt](inverted index) add inlist condition handling to compound (#34134)

2024-05-10 14:35:47 +08:00

tablet_reader.h

[Bug](top-n) do not get runtime predicate when predicate not initialized (#32208)

2024-03-15 18:06:15 +08:00

tablet_schema_cache.cpp

[fix](memory) Fix LRU cache deleter and memory tracking (#32080)

2024-03-15 17:57:58 +08:00

tablet_schema_cache.h

[fix](memory) Fix LRU cache deleter and memory tracking (#32080)

2024-03-15 17:57:58 +08:00

tablet_schema.cpp

[fix](inverted_index) donot use int32_t for index id to avoid overflow (#35062)

2024-05-21 12:58:38 +08:00

tablet_schema.h

[fix](inverted_index) donot use int32_t for index id to avoid overflow (#35062)

2024-05-21 12:58:38 +08:00

tablet.cpp

(cherry-pick)[branch-2.1] add calc tablet file crc and fix single compaction test #33076 #34915 (#35215)

2024-05-26 17:15:09 +08:00

tablet.h

(cherry-pick)[branch-2.1] add calc tablet file crc and fix single compaction test #33076 #34915 (#35215)

2024-05-26 17:15:09 +08:00

txn_manager.cpp

[fix](memory) Fix LRU cache deleter and memory tracking (#32080)

2024-03-15 17:57:58 +08:00

txn_manager.h

[fix](memory) Fix LRU cache deleter and memory tracking (#32080)

2024-03-15 17:57:58 +08:00

types.cpp

[Refactor](opentelemetry) Remove opentelemetry (#26605)

2023-11-09 18:05:34 +08:00

types.h

[bug](ipv6) the ipv6 type should be uint128_t (#34121)

2024-05-10 14:43:46 +08:00

uint24.h

…

utils.cpp

[improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library (#30573)

2024-02-05 22:00:24 +08:00

utils.h

[fix](auto inc) db_id and table_id should be int64_t instead of int32_t (#34912)

2024-05-18 18:29:59 +08:00

version_graph.cpp

[Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly (#23395)

2023-09-29 22:38:52 +08:00

version_graph.h

[Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly (#23395)

2023-09-29 22:38:52 +08:00

wrapper_field.cpp

[fix](log) regularise some BE error type and fix a load task check #28729

2023-12-25 10:45:19 +08:00

wrapper_field.h

[fix](log) regularise some BE error type and fix a load task check #28729

2023-12-25 10:45:19 +08:00