Commit Graph

948 Commits

Author SHA1 Message Date
98a1311aa2 [Opt](scanner-scheduler) Opt scanner scheduler starvation issue. (#41484)
## Proposed changes

Backport #40641
2024-09-30 15:40:20 +08:00
34429bfa0e [Chore](inverted index) remove useless code of compound filters for inverted index #40258 (#41448)
cherry pick from #40258
2024-09-29 17:27:29 +08:00
0b4552f74b [cherry-pick](branch-2.1) pick hive text write from master (#40537)
## Proposed changes
pick prs:
https://github.com/apache/doris/pull/38549
https://github.com/apache/doris/pull/40183
https://github.com/apache/doris/pull/40315

---------

Co-authored-by: Calvin Kirs <kirs@apache.org>
2024-09-27 20:57:07 +08:00
11aab648a3 [fix](crash) be crash because of duplicate __PARTIAL_UPDATE_AUTO_INC_COLUMN__ (#41172) (#41187) 2024-09-25 19:49:57 +08:00
c9d71cdc64 [Enhancement](brpc)Added enable_brpc_builtin_services parameter in be.conf (#40718) (#41047)
bp #40718 

## Proposed changes

Since [brpc's built-in
services](https://brpc.apache.org/zh/docs/builtin-services/buildin_services/)
will expose some internal server status, for security reasons, add the
parameter `enable_brpc_builtin_services` in `be.conf`. When the
parameter is false, this built-in service can be disabled.
2024-09-21 08:45:38 +08:00
49dec9f39d [branch-2.1] Picks "[opt](merge-on-write) Reduce the version not continuous logs for merge-on-write table #40946" (#40996)
picks https://github.com/apache/doris/pull/40946
2024-09-19 23:58:05 +08:00
b52b572ade [branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 (#40850)
pick
#38960
#39908
#40043
#40092
#40016
#40439

---------

Co-authored-by: hui lai <1353307710@qq.com>
Co-authored-by: yiguolei <676222867@qq.com>
2024-09-15 23:47:53 +08:00
f16615a1fc [branch-2.1](memory) Allocator support address sanitizers (#40836)
pick
#33396
#33862
#33853
#33732
#33841
#33933
#34901
#35014

---------

Co-authored-by: yiguolei <676222867@qq.com>
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-09-14 12:12:44 +08:00
c79621fff1 [enhancement](schema-change) Make the schema change memory space adaptive (#40822)
pick: #34350
2024-09-14 11:17:41 +08:00
3395cd5ce9 [PipelineX](improvement) Prepare tasks in parallel (#40270)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-09-13 13:34:29 +08:00
431e2e1af9 [Enhancement](be-logger) Support custom date time format functionality in be log. (#40727)
## Proposed changes

backport #40347
2024-09-13 10:02:12 +08:00
354967c09f [branch-2.1](memory) pick reserve memory and workload group (#40543)
1. pick #38494
2. pick #39862
3. remove vdata_stream_test, master has been removed
2024-09-09 21:16:06 +08:00
653daeb8cb Revert "[enhancement](index compaction) Enable index compaction by default (#36812)" (#40351)
Reverts apache/doris#38676
2024-09-04 14:16:09 +08:00
c8d3202595 [regression-test](cases) optimize some cases (#40240)
#40174
2024-09-02 14:50:48 +08:00
d949df0c4f [chore](cancel) Make enable_pipeline_task_leakage_detect false by default #40145 (#40192)
cherry pick from #40145
2024-08-30 22:48:00 +08:00
ca07a00c93 Revert "[branch-2.1](hive) support hive write text table (#38549) (#4… (#40157)
…0063)"

This reverts commit c6df7c21a3c09ae1664deabacb88dfcea9d94b68.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-08-30 10:25:38 +08:00
c6df7c21a3 [branch-2.1](hive) support hive write text table (#38549) (#40063)
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd

pick from https://github.com/apache/doris/pull/38549
2024-08-29 16:50:40 +08:00
6915d76731 [opt](file-cache) add evict file number per round (#39721)
Previously, when getting block from file cache, it may try to evict
lots of blocks to reserve capacity for lru cache. This operation may
take long time
while hold the lock, causing other operation blocked.

This PR add a new BE config `file_cache_max_evict_num_per_round`,
default is 1000, so that it will not hold lock for a long time.
2024-08-28 08:49:12 +08:00
cb312cabb2 [Fix](tablet-meta) limit the data size of tablet meta (#39455) (#39974)
pick master #39455
2024-08-27 20:10:17 +08:00
460605ae3c [branch-2.1] pick some prs (#39860)
## Proposed changes

Issue Number: close #xxx

https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
2024-08-24 17:26:42 +08:00
ae4d747c13 [branch-2.1](memory) Modify memory gc conf and add crash_in_alloc_large_memory_bytes (#39834)
pick #39611
2024-08-24 09:21:35 +08:00
9d5468d198 [branch-2.1](memory) BE memory info compatible with CgroupV2 (#39799)
pick #39256
2024-08-23 02:03:00 +08:00
1367f74e7a [branch-2.1](memory) Optimize ClearCacheActionimplementation (#39796)
pick #38438
2024-08-23 01:51:14 +08:00
13b882a4cc [branch-2.1](memory) Add memory metrics to bvar (#39763)
pick #38391
2024-08-22 17:34:30 +08:00
8ce8887b75 [branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth (#39760)
pick #38168
overwrites changes in #37221 on workload_group_manager.cpp. If need to
pick 37221, ignore it.
2024-08-22 17:33:11 +08:00
610f69432a [improvement](segmentcache) limit segment cache by fd limit or memory… (#39689)
… (#39658)

remove a useless config.
2024-08-21 15:19:52 +08:00
830f250a80 [opt](query cancel) cancel query if it has pipeline task leakage #39223 (#39537)
pick #39223 with some modifications. Optimization will only be applied
to pipeline x.
2024-08-19 14:33:59 +08:00
0680c8d314 [improve](cache) File cache async init (#39036)
## Proposed changes

Do `load_cache_info_into_memory()` asynchronously in a background thread
in `LRUFileCache::initialize()`.
When the cache is not ready, `LRUFileCache::get_or_set()` will return
the FileBlock which state is SKIP_CACHE.
2024-08-15 16:27:51 +08:00
b38caed808 [Improve](columns)replace fatal with exception #38035 (#38996) 2024-08-12 09:51:30 +08:00
8678fcea32 [config](inverted index)Make inverted_index_ram_dir enable by default(#35094) (#39120)
## Proposed changes

bp #35094

Co-authored-by: Luennng <luennng@gmail.com>
2024-08-09 01:38:14 +08:00
7e95d7cbec [bugfix](backup)(cooldown) cancel backup properly when be backup failed (#38724) (#38993)
Co-authored-by: zhangyuan <ayuanzhang@tencent.com>
2024-08-07 15:58:11 +08:00
2543b569bb [Optimize](Row store) pick #37145, #38236 (#38932) 2024-08-07 09:55:42 +08:00
e9bf0776d7 [fix](parquet) disable parquet page index by default #38691 (#38901)
bp #38691
2024-08-06 08:51:39 +08:00
0603ec1d9d [enhancement](compaction) optimizing memory usage for compaction (#37099) (#37486) 2024-08-04 10:49:18 +08:00
b3f335ba5f [enhancement](index compaction) Enable index compaction by default (#36812) (#38676)
## Proposed changes

bp #36812
2024-08-02 12:03:57 +08:00
0152a4e86f [config](be) add be config migration_lock_timeout_ms (#38000) (#38337)
backport #38000
2024-07-25 17:36:34 +08:00
10c5c336d8 [branch-2.1](arrow-flight-sql) Add config arrow_flight_result_sink_buffer_size_rows (#38223)
pick #38221
2024-07-24 15:15:39 +08:00
7b141ffde7 [pick]add min scan thread num for workload group's scan thread (#38123)
## Proposed changes

pick #38096
2024-07-19 18:43:05 +08:00
b15ccdbe98 [Pick](Variant) pick some fix (#37922)
#37674
#37839
#37883 
#37857 
#37794
2024-07-16 21:38:47 +08:00
9861f81630 [branch-2.1](memory) Fix Jemalloc Cache Memory Tracker (#37905)
pick #37464
2024-07-16 19:01:31 +08:00
47096f2083 [test](regression) add cases for data quality error url (#34987) (#37777)
cherry-pick #34987
2024-07-16 11:12:52 +08:00
Pxl
010d9d88f8 [Feature](rpc) support set brpc_idle_timeout_sec and enable thrift so… (#37808)
pick from #37333
2024-07-15 21:12:25 +08:00
a4d37d96ca [opt](file-scanner) add not found file number in profile (#37042) (#37764)
bp #37042
2024-07-15 17:11:06 +08:00
232202b71f [improve](load) reduce memory reserved in memtable limiter (#37511) (#37699)
cherry-pick #37511
2024-07-15 11:09:09 +08:00
2759383365 [branch-2.1](timezone) refactor tzdata load to accelerate and unify timezone parsing (#37062) (#37269)
pick https://github.com/apache/doris/pull/37062

1. revert https://github.com/apache/doris/pull/25097. we decide to rely
on OS. not maintain independent tzdata anymore to keep result
consistency
2. refactor timezone load. removed rwlock.

before:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (6.88 sec)
```
now:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (2.61 sec)
```
3. now don't support timezone offset format string like 'UTC+8', like we
already said in
https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage
4. support case-insensitive timezone parsing in nereids.
5. a bug when parse timezone using nereids. should check DST by input,
but wrongly by now before. now fixed.

doc pr: https://github.com/apache/doris-website/pull/810
2024-07-15 10:56:48 +08:00
747172237a [branch-2.1](memory) Pick some memory GC patch (#37725)
pick
#36768
#37164
#37174
#37525
2024-07-14 15:19:40 +08:00
a61030215e [branch-2.1](memory) Support make all memory snapshots (#37705)
pick #36679
2024-07-12 16:21:37 +08:00
ef031c5fb2 [branch-2.1](memory) Fix reserve memory compatible with memory GC and logging (#37682)
pick
#36307
#36412
2024-07-12 11:43:26 +08:00
cf2fb6945a [branch-2.1](memory) Refactor LRU cache policy memory tracking (#37658)
pick 
#36235
#35965
2024-07-11 21:04:01 +08:00
3337c1bbe3 [[enhancement](compaction) adjust compaction concurrency based on compaction score and workload (#37491)
adjust compaction concurrency based on compaction score and workload
#36672
fix null pointer when retrieving CPU load average #37171
2024-07-09 09:56:35 +08:00