df26475e1a
[Enhancement](compaction) enable the compaction producer to generate multiple compaction tasks in a single run ( #45411 ) ( #46160 )
...
pick master #45411
2024-12-31 09:51:43 +08:00
1d742b5f7d
[Cherry-pick](branch-2.1) Pick "[Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset ( #43466 )" ( #46117 )
...
Before this PR, in cases where there is an alternating distribution of
data rowset -> delete rowset -> data rowset -> delete rowset, cumulative
compaction would only move the cumulative point forward to allow base
compaction to handle the delete rowset. Cumulative compaction itself
would not process the data and would return be marked as failure. This
would cause the compaction submission task process to pause for 5
seconds, impacting efficiency.
This PR modifies the return status to OK for such cases, which improves
the efficiency of the compaction submission task.
2024-12-30 10:18:57 +08:00
44fa86040f
branch-2.1: [Enhancement](compaction)Optimize compaction task permit allocation #45197 ( #45243 )
...
Cherry-picked from #45197
Co-authored-by: abmdocrt <lianyukang@selectdb.com >
2024-12-11 14:35:07 +08:00
d0737c45f8
[cherry-pick](branch-2.1) Pick "[Fix](Rowset Id) Use a randomly generated rowset ID to handle memory write failures ( #42949 )" ( #44086 )
2024-12-10 15:48:02 +08:00
6714936f8b
[pick](branch-2.1) pick #39962 #40304 ( #44931 )
2024-12-04 17:56:58 +08:00
6726c9bf2f
[improvement](compaction) reduce tablet skip compaction time ( #44273 ) ( #44791 )
...
pick master #44273
The time for tablet skip compaction is 120 seconds, which is too long.
In the scenario of high-frequency import (mow), it leads to a high
compaction score. Therefore, reducing the skip time to 10 seconds is
necessary.
2024-12-02 10:07:17 +08:00
071a2106a3
branch-2.1: [opt](brpc) Make the timeout of brpc checking configurable #44481 ( #44602 )
...
Cherry-picked from #44481
Co-authored-by: Jerry Hu <hushenggang@selectdb.com >
2024-11-26 19:26:39 +08:00
83208ee1a8
[pick](branch-2.1) pick #43960 #43929 #44177 ( #44240 )
...
pick #43960 #43929 #44177
2024-11-19 17:25:16 +08:00
ea61206233
[pick](branch-2.1) pick #43281 ( #44020 )
...
pick #43281
2024-11-16 21:53:21 +08:00
7fc78e3f87
[opt](brpc) check and remove unavailable brpc stubs ( #43212 ) ( #43859 )
2024-11-14 19:52:06 +08:00
c3878167a5
[fix](config) change segment_cache_fd_percentage and inverted_index_fd_number_limit_percent default value #42224 ( #42423 )
2024-10-28 09:24:59 +08:00
e6bc8ab79e
[Opt](config) adjust segment cache ( #42308 ) ( #42431 )
...
#42308
2024-10-24 23:05:36 +08:00
968e33f07e
[cherry-pick](branch-21) pick ( #39057 ) ( #41352 ) ( #41958 )
...
## Proposed changes
pick from master (#39057 ) (#41352 )
<!--Describe your changes.-->
---------
Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com >
2024-10-17 14:30:40 +08:00
e56216211e
[pick](branch-2.1) pick #40667 #40714 ( #41905 )
...
pick
#40667
#40714
---------
Co-authored-by: wangbo <wangbo@apache.org >
2024-10-16 14:09:03 +08:00
98a1311aa2
[Opt](scanner-scheduler) Opt scanner scheduler starvation issue. ( #41484 )
...
## Proposed changes
Backport #40641
2024-09-30 15:40:20 +08:00
34429bfa0e
[Chore](inverted index) remove useless code of compound filters for inverted index #40258 ( #41448 )
...
cherry pick from #40258
2024-09-29 17:27:29 +08:00
0b4552f74b
[cherry-pick](branch-2.1) pick hive text write from master ( #40537 )
...
## Proposed changes
pick prs:
https://github.com/apache/doris/pull/38549
https://github.com/apache/doris/pull/40183
https://github.com/apache/doris/pull/40315
---------
Co-authored-by: Calvin Kirs <kirs@apache.org >
2024-09-27 20:57:07 +08:00
c9d71cdc64
[Enhancement](brpc)Added enable_brpc_builtin_services parameter in be.conf ( #40718 ) ( #41047 )
...
bp #40718
## Proposed changes
Since [brpc's built-in
services](https://brpc.apache.org/zh/docs/builtin-services/buildin_services/ )
will expose some internal server status, for security reasons, add the
parameter `enable_brpc_builtin_services` in `be.conf`. When the
parameter is false, this built-in service can be disabled.
2024-09-21 08:45:38 +08:00
49dec9f39d
[branch-2.1] Picks "[opt](merge-on-write) Reduce the version not continuous logs for merge-on-write table #40946 " ( #40996 )
...
picks https://github.com/apache/doris/pull/40946
2024-09-19 23:58:05 +08:00
b52b572ade
[branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 ( #40850 )
...
pick
#38960
#39908
#40043
#40092
#40016
#40439
---------
Co-authored-by: hui lai <1353307710@qq.com >
Co-authored-by: yiguolei <676222867@qq.com >
2024-09-15 23:47:53 +08:00
f16615a1fc
[branch-2.1](memory) Allocator support address sanitizers ( #40836 )
...
pick
#33396
#33862
#33853
#33732
#33841
#33933
#34901
#35014
---------
Co-authored-by: yiguolei <676222867@qq.com >
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-09-14 12:12:44 +08:00
c79621fff1
[enhancement](schema-change) Make the schema change memory space adaptive ( #40822 )
...
pick: #34350
2024-09-14 11:17:41 +08:00
3395cd5ce9
[PipelineX](improvement) Prepare tasks in parallel ( #40270 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-13 13:34:29 +08:00
431e2e1af9
[Enhancement](be-logger) Support custom date time format functionality in be log. ( #40727 )
...
## Proposed changes
backport #40347
2024-09-13 10:02:12 +08:00
653daeb8cb
Revert "[enhancement](index compaction) Enable index compaction by default ( #36812 )" ( #40351 )
...
Reverts apache/doris#38676
2024-09-04 14:16:09 +08:00
c8d3202595
[regression-test](cases) optimize some cases ( #40240 )
...
#40174
2024-09-02 14:50:48 +08:00
d949df0c4f
[chore](cancel) Make enable_pipeline_task_leakage_detect false by default #40145 ( #40192 )
...
cherry pick from #40145
2024-08-30 22:48:00 +08:00
ca07a00c93
Revert "[branch-2.1](hive) support hive write text table ( #38549 ) (#4… ( #40157 )
...
…0063)"
This reverts commit c6df7c21a3c09ae1664deabacb88dfcea9d94b68.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-08-30 10:25:38 +08:00
c6df7c21a3
[branch-2.1](hive) support hive write text table ( #38549 ) ( #40063 )
...
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd
pick from https://github.com/apache/doris/pull/38549
2024-08-29 16:50:40 +08:00
6915d76731
[opt](file-cache) add evict file number per round ( #39721 )
...
Previously, when getting block from file cache, it may try to evict
lots of blocks to reserve capacity for lru cache. This operation may
take long time
while hold the lock, causing other operation blocked.
This PR add a new BE config `file_cache_max_evict_num_per_round`,
default is 1000, so that it will not hold lock for a long time.
2024-08-28 08:49:12 +08:00
cb312cabb2
[Fix](tablet-meta) limit the data size of tablet meta ( #39455 ) ( #39974 )
...
pick master #39455
2024-08-27 20:10:17 +08:00
ae4d747c13
[branch-2.1](memory) Modify memory gc conf and add crash_in_alloc_large_memory_bytes ( #39834 )
...
pick #39611
2024-08-24 09:21:35 +08:00
1367f74e7a
[branch-2.1](memory) Optimize ClearCacheActionimplementation ( #39796 )
...
pick #38438
2024-08-23 01:51:14 +08:00
8ce8887b75
[branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth ( #39760 )
...
pick #38168
overwrites changes in #37221 on workload_group_manager.cpp. If need to
pick 37221, ignore it.
2024-08-22 17:33:11 +08:00
610f69432a
[improvement](segmentcache) limit segment cache by fd limit or memory… ( #39689 )
...
… (#39658 )
remove a useless config.
2024-08-21 15:19:52 +08:00
830f250a80
[opt](query cancel) cancel query if it has pipeline task leakage #39223 ( #39537 )
...
pick #39223 with some modifications. Optimization will only be applied
to pipeline x.
2024-08-19 14:33:59 +08:00
0680c8d314
[improve](cache) File cache async init ( #39036 )
...
## Proposed changes
Do `load_cache_info_into_memory()` asynchronously in a background thread
in `LRUFileCache::initialize()`.
When the cache is not ready, `LRUFileCache::get_or_set()` will return
the FileBlock which state is SKIP_CACHE.
2024-08-15 16:27:51 +08:00
8678fcea32
[config](inverted index)Make inverted_index_ram_dir enable by default( #35094 ) ( #39120 )
...
## Proposed changes
bp #35094
Co-authored-by: Luennng <luennng@gmail.com >
2024-08-09 01:38:14 +08:00
2543b569bb
[Optimize](Row store) pick #37145 , #38236 ( #38932 )
2024-08-07 09:55:42 +08:00
e9bf0776d7
[fix](parquet) disable parquet page index by default #38691 ( #38901 )
...
bp #38691
2024-08-06 08:51:39 +08:00
0603ec1d9d
[enhancement](compaction) optimizing memory usage for compaction ( #37099 ) ( #37486 )
2024-08-04 10:49:18 +08:00
b3f335ba5f
[enhancement](index compaction) Enable index compaction by default ( #36812 ) ( #38676 )
...
## Proposed changes
bp #36812
2024-08-02 12:03:57 +08:00
0152a4e86f
[config](be) add be config migration_lock_timeout_ms ( #38000 ) ( #38337 )
...
backport #38000
2024-07-25 17:36:34 +08:00
10c5c336d8
[branch-2.1](arrow-flight-sql) Add config arrow_flight_result_sink_buffer_size_rows ( #38223 )
...
pick #38221
2024-07-24 15:15:39 +08:00
7b141ffde7
[pick]add min scan thread num for workload group's scan thread ( #38123 )
...
## Proposed changes
pick #38096
2024-07-19 18:43:05 +08:00
b15ccdbe98
[Pick](Variant) pick some fix ( #37922 )
...
#37674
#37839
#37883
#37857
#37794
2024-07-16 21:38:47 +08:00
9861f81630
[branch-2.1](memory) Fix Jemalloc Cache Memory Tracker ( #37905 )
...
pick #37464
2024-07-16 19:01:31 +08:00
010d9d88f8
[Feature](rpc) support set brpc_idle_timeout_sec and enable thrift so… ( #37808 )
...
pick from #37333
2024-07-15 21:12:25 +08:00
a4d37d96ca
[opt](file-scanner) add not found file number in profile ( #37042 ) ( #37764 )
...
bp #37042
2024-07-15 17:11:06 +08:00
232202b71f
[improve](load) reduce memory reserved in memtable limiter ( #37511 ) ( #37699 )
...
cherry-pick #37511
2024-07-15 11:09:09 +08:00
