Commit Graph

7428 Commits

Author SHA1 Message Date
3c9c6c18a8 [Enhancement](hive-writer) Write only regular fields to file in the hive-writer. (#33000) 2024-04-12 10:29:08 +08:00
3d66723214 [branch-2.1](auto-partition) pick auto partition and some more prs (#33523) 2024-04-11 17:12:17 +08:00
af95302088 fix compile 2024-04-11 13:10:24 +08:00
69fc8cf06d [branch-2.1](memory) Fix rowid storage reader memory tracker (#33521)
fix:
F20240411 10:26:06.693233 1368925 thread_context.h:293] __builtin_unreachable, If you crash here, it means that SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not used correctly. starting position of each thread is expected to use SCOPED_ATTACH_TASK to bind a MemTrackerLimiter belonging to Query/Load/Compaction/Other Tasks, otherwise memory alloc using Doris Allocator in the thread will crash. If you want to switch MemTrackerLimiter during thread execution, please use SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER, do not repeat Attach. Of course, you can modify enable_memory_orphan_check=false in be.conf to avoid this crash.
*** Check failure stack trace: ***
    @     0x562d9b5aa6a6  google::LogMessage::SendToLog()
    @     0x562d9b5a70f0  google::LogMessage::Flush()
    @     0x562d9b5aaee9  google::LogMessageFatal::~LogMessageFatal()
    @     0x562d7ebd1b7e  doris::thread_context()
    @     0x562d7ec203b8  Allocator<>::sys_memory_check()
    @     0x562d7ec255a3  Allocator<>::memory_check()
    @     0x562d7ec274a1  Allocator<>::alloc_impl()
    @     0x562d7ec27227  Allocator<>::alloc()
    @     0x562d67a12207  doris::vectorized::PODArrayBase<>::alloc<>()
    @     0x562d67a11fde  doris::vectorized::PODArrayBase<>::realloc<>()
    @     0x562d67a11e26  doris::vectorized::PODArrayBase<>::reserve<>()
    @     0x562d77331ee3  doris::vectorized::ColumnVector<>::reserve()
    @     0x562d7e64328e  doris::vectorized::ColumnNullable::reserve()
    @     0x562d7ec79a84  doris::vectorized::Block::Block()
    @     0x562d6b86b81b  doris::PInternalServiceImpl::_multi_get()
    @     0x562d6b8a4a07  doris::PInternalServiceImpl::multiget_data()::$_0::operator()()
2024-04-11 13:10:24 +08:00
Pxl
5688c28364 [Bug](runtime-filter) try to fix heap use after free on runtime filter send filter size (#33465) (#33522) 2024-04-11 13:10:24 +08:00
f7d52b5b1c [feature](expr) add type check when expr prepare (#33330) 2024-04-11 09:31:50 +08:00
bc929686e3 [feature](debug point) add macro DBUG_RUN_CALLBACK (#33407) 2024-04-11 09:31:50 +08:00
ef26479282 [improve](serde) support complex type in write/read pb serde (#33124)
support complex type and ip/jsonb in DataTypeSerDe::write_column_to_pb/read_column_from_pb function
2024-04-11 09:31:50 +08:00
ea1e542e31 [fix](partial-update) remove unnecessary DECHEK on IndexChannel::num_rows_filtered (#33160) 2024-04-11 09:31:50 +08:00
Pxl
3081fc584d [Improvement](runtime-filter) support sync join node build side's size to init bloom runtime filter (#32180)
support sync join node build side's size to init bloom runtime filter
2024-04-11 09:31:50 +08:00
58347aa2b7 [branch-2.1](bugfix) fix alter tablet not have memtracker
[branch-2.1](memory) Fix EngineAlterTabletTask attach memory tracker #33506
2024-04-11 09:29:42 +08:00
3e7b253e41 [fix](compress) data decompress failed while max_len equals 8M (#33456)
* fix data decompress failed while max_len equals MAX_COMPRESSION_BUFFER_SIZE_FOR_REUSE

* update code format error
2024-04-11 08:36:44 +08:00
f3a60374b5 [branch-2.1](memory) Fix EngineStorageMigrationTask MemTracker attach thread (#33509) 2024-04-11 08:35:13 +08:00
6bef95eb4f [fix](memory) Fix memory tracker destructor deadlock (#33497) 2024-04-10 22:46:53 +08:00
6bf1fa4b63 [branch-2.1](memory) Fix SnapshotManager MemTracker attach thread #33501 2024-04-10 22:22:12 +08:00
90ee9c171e [bug](mem_tracker) fix mem_tracker dcheck failed as not used correctly (#33349) (#33486)
* [bug](mem_tracker) fix mem_tracker dcheck failed as not used correctly
2024-04-10 22:22:12 +08:00
e2ad7149c3 [feature](debug point) Add handler to debug point (#33350) 2024-04-10 16:24:13 +08:00
53505e903b [scan](improvement) Adjust parallel scanners num (#33223) 2024-04-10 16:23:20 +08:00
0e262ba0e4 [improvement](spill) improve cancel of spill and improve log printing (#33229)
* [improvement](spill) improve cancel of spill and improve log printing

* fix
2024-04-10 16:23:20 +08:00
28acfaed2b [fix](pipeline)group by and output is empty (#33192) 2024-04-10 16:23:20 +08:00
d667df2d06 [improvement](spill) avoid unnecessary spilling in hash join build phase (#33277) 2024-04-10 16:21:50 +08:00
5e73d7a281 [fix](compaction) fix incorrect grouping of vertical compaction columns in tables only with key columns (#32896) (#33470) 2024-04-10 16:04:33 +08:00
f8d1fa2be3 [chore](multi-table-load) add context info in log when using single-stream-multi-table load (#33317) 2024-04-10 16:03:05 +08:00
Pxl
6412753517 [improve](exec) reduce copy on store_string_ref (#33232) 2024-04-10 16:00:12 +08:00
e6e2099256 [fix](spill) fix hash join error 'invalid slot id' (#33273) 2024-04-10 16:00:12 +08:00
7fae123b01 [FIX](inverted_index) fix inverted index write array with _doc is empty (#33170) 2024-04-10 15:26:09 +08:00
2b1ab89b5b [fix](memory) Fix memory log compile by ASAN (#33162)
ASAN compiles BE, add markers in memory logs
2024-04-10 15:26:09 +08:00
9670422d61 [fix](inverted index) fix the incorrect result issue of COUNT_ON_INDEX for key columns (#33164) 2024-04-10 15:26:09 +08:00
2e40e39584 [chore](spill) add timers for performance tuning (#33185) 2024-04-10 15:26:08 +08:00
Pxl
8fd6d4c41b [Chore](build) add -Wconversion and remove some unused code (#33127)
add -Wconversion and remove some unused code
2024-04-10 15:26:08 +08:00
cc363f26c2 [fix](Nereids) fix group concat (#33091)
Fix failed in regression_test/suites/query_p0/group_concat/test_group_concat.groovy

select
group_concat( distinct b1, '?'), group_concat( distinct b3, '?')
from
table_group_concat
group by
b2

exception:

lowestCostPlans with physicalProperties(GATHER) doesn't exist in root group

The root cause is '?' is push down to slot by NormalizeAggregate, AggregateStrategies treat the slot as a distinct parameter and generate a invalid PhysicalHashAggregate, and then reject by ChildOutputPropertyDeriver.

I fix this bug by avoid push down literal to slot in NormalizeAggregate, and forbidden generate stream aggregate node when group by slots is empty
2024-04-10 14:59:46 +08:00
6c5dd820c0 [improvement](spill) improve spill timers (#33156) 2024-04-10 14:55:11 +08:00
7f2fdf78ac [Enhancement](inverted index) set need to read data only when delete predicate contains the column (#33172) 2024-04-10 14:53:56 +08:00
c61d6ad1e2 [Feature] support function uuid_to_int and int_to_uuid #33005 2024-04-10 14:53:56 +08:00
bf022f9d8d [enhancement](function truncate) truncate can use column as scale argument (#32746)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-10 14:53:56 +08:00
8b1d174b13 [Optimize] Move strings_pool from individual tree nodes to the tree itself (#33089)
Previously, strings_pool was allocated within each tree node. However, due to the Arena's alignment of allocated chunks to at least 4K, this allocation size was excessively large for a single tree node. Consequently, when there are numerous nodes within the SubcolumnTree, a significant portion of memory was wasted. Moving strings_pool to the tree itself optimizes memory usage and reduces wastage, improving overall efficiency.
2024-04-10 14:53:56 +08:00
1b3a11a02b [Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)
* The default delete bitmap cache is set to 100MB, which can be insufficient and cause performance issues when the amount of user data is large. To mitigate the problem of an inadequate cache, we will take the larger of 5% of the total memory and 100MB as the delete bitmap cache size.
2024-04-10 14:53:56 +08:00
3b42dc73af [improvement](spill) avoid spill if memory is enough (#33075) 2024-04-10 14:53:27 +08:00
517c12478f [improvement](spill) spill trigger improvement (#32641) 2024-04-10 14:52:46 +08:00
b0b5f84e40 [feature](load) support compressed JSON format data for broker load (#30809) 2024-04-10 14:20:53 +08:00
1d0908e80d [feature](profile) make WaitForLocalExchangeBuffer timer merge (#32946)
make WaitForLocalExchangeBuffer timer merge
2024-04-10 11:57:57 +08:00
Pxl
e4993a19e5 [Chore](column) remove ColumnVectorHelper (#33036)
remove ColumnVectorHelper
2024-04-10 11:56:41 +08:00
8e19cdd745 [featrue](expr) support common subexpression elimination be part (#32673) 2024-04-10 11:56:21 +08:00
5116724494 [Fix](hive-writer) Fix the issue of block was not copied to do filtering when hive partition writer write block to file. (#32775) (#33447)
backport #32775
2024-04-10 11:42:23 +08:00
4963d60a07 [Fix](multi-catalog)Fix the issue of not initializing the writer caused by refactoring and add hive writing regression test. (#32721) (#33446)
backport #32721.
2024-04-10 11:42:22 +08:00
cf7595d423 [opt](memory) Optimize mem tracker accuracy (#32039) (#33140) 2024-04-10 11:42:19 +08:00
39fba884fb [fix](typo) typo fix for 'delete bimap' changing to 'delete bitmap' (#32341) 2024-04-10 11:34:30 +08:00
3243053fcd [fix](memory) Fix MemTableWriter flush_async attach task in thread context (#33071) 2024-04-10 11:34:30 +08:00
285e2fcb5a [fix] (vectorization) regexp all_pass string (#32515) 2024-04-10 11:34:30 +08:00
c5a3af5c27 [partitionsort](fix) Fix DCHECK failure (#33035) 2024-04-10 11:34:30 +08:00