Commit Graph

8711 Commits

Author SHA1 Message Date
c8ccada010 branch-2.1: [Fix](memory) Add try catch block for Segment::load_pk_index_and_bf… (#47742)
pick https://github.com/apache/doris/pull/47715
2025-02-11 16:56:13 +08:00
9eb6c2fa60 [fix](jni) pass krb5 conf to jni (#47679)
### What problem does this PR solve?

Related PR: #47299

Problem Summary:

This pick part of #47299, only related to `jni-util.cpp`.
To pass the krb5.conf config to Jni env
2025-02-10 22:27:11 +08:00
470030b878 [feat](clone) Speed clone tablet via batch small file downloading #45061 (#45218)
cherry pick from #45061
2025-02-10 19:38:40 +08:00
f4e92d75f5 [branch-2.1] fix select * from variables system table (#34529) (#47660)
pick #34529

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
Co-authored-by: xueweizhang <zxw520blue1@163.com>
2025-02-09 14:14:06 +08:00
e47b398123 branch-2.1: [chore](binlog) Add download binlog related configs to BE #47412 (#47585)
cherry pick from #47412
2025-02-08 16:23:23 +08:00
e681c055a4 [chore](binlog) Log download binlog cost (#47360) (#47575)
cherry-pick #47360
2025-02-08 16:22:23 +08:00
b4f00be528 branch-2.1: [fix](restore) Make the DirMoveTask idempotent. #47313 (#47584)
cherry pick from #47313
2025-02-08 14:28:38 +08:00
3ec723f2cb branch-2.1: [fix](prepared statement) fix protocol with TIME datatype #47389 (#47543)
Cherry-picked from #47389

Co-authored-by: lihangyu <lihangyu@selectdb.com>
2025-02-08 13:00:49 +08:00
9d3ee27e55 branch-2.1: [fix](restore) Add a local snapshot lock to protect snapshot dir #47279 (#47574)
cherry pick from #47279
2025-02-08 09:49:22 +08:00
0cd2557882 branch-2.1: [Fix]Fix double-free when memtable agg #47523 (#47532)
Cherry-picked from #47523

Co-authored-by: wangbo <wangbo@selectdb.com>
2025-02-06 13:11:05 +08:00
209ddb374e branch-2.1: [chore](io) Add debug log for critical file operations #46770 (#46859)
cherry pick from #46770
2025-02-06 09:41:45 +08:00
Pxl
631d1f43cf [Chore](profile) add some profile on ReaderInit #45556 (#47427)
pick from #45556
2025-01-26 10:51:08 +08:00
2ae03d7e02 branch-2.1: [Bug](function) fix wrong result when case when have more than 256 conditions and the… #47179 (#47392)
Cherry-picked from #47179

Co-authored-by: Pxl <xl@selectdb.com>
2025-01-25 10:04:50 +08:00
dd7affb8f9 branch-2.1 [Fix](ShortCircuite) fix point query crash with prepared statement when encounter delete sign (#47227)
cherry-pick from #47178
2025-01-25 08:33:06 +08:00
3a9612d97e [Bug](fix) Try to prevent the bug of brpc issue 2146 (#47432) 2025-01-24 21:38:08 +08:00
3b7cc67ec2 branch-2.1: [fix](func) Fix precision loss in ST_GeometryFromWKB coordinate parsing #46661 (#47263)
Cherry-picked from #46661

Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com>
2025-01-24 17:34:37 +08:00
701aec6b21 branch-2.1: [opt](jsonb) add ut for the jsonb parser #47181 (#47388)
Cherry-picked from #47181

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
2025-01-24 17:29:33 +08:00
4f9562650d Branch-2.1 [Fix](Variant) fix variant serialize to string (#47121) (#47147)
cherry-pick from #47121
2025-01-18 09:12:39 +08:00
9dffe5992e branch-2.1: [fix](json) fix parsing double in jsonb #46977 (#47065)
Cherry-picked from #46977

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
2025-01-18 00:25:13 +08:00
adc9e9a161 [fix](function) Fix undefined behavior caused by bit_shift_left exceeding the 64-bit range. (#47080)
### What problem does this PR solve?
It is part of this PR's https://github.com/apache/doris/pull/35769
content.


```
SELECT a, b, bit_shift_left(nullable(a), nullable(b)) from t01;
+----------------------+-----+-----------------------------------------------------------+
| a                    | b   | bit_shift_left(nullable(a), cast(nullable(b) as TINYINT)) |
+----------------------+-----+-----------------------------------------------------------+
| -4388843650233597359 | 127 | -9223372036854775808                                      |
| 306567258363819813   | 3   | 2452538066910558504                                       |
| 306567258363819813   | 127 | -9223372036854775808                                      |
| 3456789123456789123  | 127 | -9223372036854775808                                      |
+----------------------+-----+-----------------------------------------------------------+ 
```
now
```
SELECT a, b, bit_shift_left(nullable(a), nullable(b)) from t01;
+----------------------+-----+-----------------------------------------------------------+
| a                    | b   | bit_shift_left(nullable(a), cast(nullable(b) as TINYINT)) |
+----------------------+-----+-----------------------------------------------------------+
| -4388843650233597359 | 127 | 0                                                         |
| 306567258363819813   | 3   | 2452538066910558504                                       |
| 306567258363819813   | 127 | 0                                                         |
| 3456789123456789123  | 127 | 0                                                         |
+----------------------+-----+-----------------------------------------------------------+
```
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-01-16 21:13:24 +08:00
Pxl
a85b5eb9d0 [Bug](runtime-filter) fix bloom_filter_func use after free on get_build_bf_cardinality #47034 (#47050)
pick from #47034
2025-01-16 14:22:36 +08:00
Pxl
05e0bff137 [Bug](scan) do not ignore non-rf predicate which acting same column with rf #46789 (#47015)
pick from #46789
2025-01-15 16:38:35 +08:00
c6ffa1f4eb [Fix](PreparedStatement) fix incorrect query result with prepared statment (#47012)
cherry-pick from #46994
2025-01-15 14:42:04 +08:00
664bbeb10c [fix](data stream) Fix dead loop in VDataStreamMgr's de-constructor (… (#47000) 2025-01-15 13:53:20 +08:00
Pxl
b1a2846b78 [Chore](schema-change) return error when tablet_schema get null function #46790 (#46962)
pick from #46790
2025-01-14 15:27:17 +08:00
Pxl
58415c3591 [Chore](case) add test case for cityhash #46928 (#46957)
pick from #46928
2025-01-14 14:03:19 +08:00
Pxl
989f189486 [test](join) Fuzzy disable runtime filters in BE #45654 (#46931)
pick from  #45654

---------

Co-authored-by: Jerry Hu <hushenggang@selectdb.com>
2025-01-14 12:46:56 +08:00
a12f4d7a09 [fix](pipelineX) bp #46493 and add some regresstion tests (#46865)
### What problem does this PR solve?

bp #46493
Add some regresstion tests for #46493 and #46541

Co-authored-by: liulijia <liulijia1029@gmail.com>
2025-01-14 10:11:58 +08:00
ae9d3391be branch-2.1: [improve](move-memtable) reduce flush token num #46001 (#46926)
cherry pick from #46001

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
2025-01-14 09:44:30 +08:00
7d35838f4f [branch-2.1]Refactor workload group metric (#46640) (#46875)
pick #46640
2025-01-13 15:16:53 +08:00
fa60c60b42 [fix](pipelien) should not finalize probe when wake up early in SetProbeSinkOperatorX (#46706) (#46832)
```
*** Query id: 80819fcc223e4a45-b46155de6e0c4eee ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1736352810 (unix time) try "date -d @1736352810" if you are using GNU date ***
*** Current BE git commitID: 08683cbaf5 ***
*** SIGSEGV address not mapped to object (@0x38) received by PID 8736 (TID 11549 OR 0x7f8dd0922640) from PID 56; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F92019CA520 in /lib/x86_64-linux-gnu/libc.so.6
 4# auto doris::pipeline::SetProbeSinkOperatorX::_refresh_hash_table(doris::pipeline::SetProbeSinkLocalState&)::{lambda(auto:1&&)#1}::operator(), HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&>(doris::vectorized::MethodSerialized, HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&) const at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:213
 5# doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:184
 6# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:98
 7# doris::pipeline::PipelineTask::execute(bool*) at /root/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:387
 8# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:138
 9# doris::ThreadPool::dispatch_thread() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
10# doris::Thread::supervise_thread(void*) at /root/doris_branch-3.0/doris/be/src/util/thread.cpp:499
11# start_thread at ./nptl/pthread_create.c:442
12# 0x00007F9201AAE850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```
2025-01-13 14:21:43 +08:00
9ccfff12ab [pick](branch-2.1) pick #46266 (#46840)
### What problem does this PR solve?

```
[fix](cancel) Fix BufferControlBlock cancel msg #46266
```
2025-01-13 10:37:11 +08:00
94856b19a2 [fix](hive)fix select count(*) hive full acid tb opt error. (#46732) (#46804)
bp #46732
2025-01-12 08:55:10 +08:00
42808a02dc [Fix](func) fix group_array_intersect result error in all arr contain null (#46590) 2025-01-10 20:14:23 +08:00
9958eb91a3 branch-2.1: [fix](compaction) fix compaction producer hold for permits leak #45664 (#46753)
Cherry-picked from #45664

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: garenshi <garenshi@tencent.com>
Co-authored-by: camby <cambyzhu@tencent.com>
2025-01-10 15:53:01 +08:00
62ad145982 branch-2.1: [fix] Add initialization list to the copy constructor of ScanPredicate in jni_connect.h #46666 (#46725)
Cherry-picked from #46666

Co-authored-by: 神技圈子 <songguangfan@gmail.com>
2025-01-10 10:48:26 +08:00
5e6669ff08 branch-2.1: [Optimize](TabletMeta) Reuse TabletSchema in TabletMeta #46572 (#46687)
Cherry-picked from #46572

Co-authored-by: lihangyu <lihangyu@selectdb.com>
2025-01-09 22:20:31 +08:00
Pxl
521e653631 [Chore](check) add error check for agg state with some function use special serializ… (#46552) 2025-01-09 11:10:44 +08:00
0ecb84c638 branch-2.1: [Fix](partial update) Fix incorrect result when partial update include delete sign columns #46194 (#46543)
pick https://github.com/apache/doris/pull/46194
2025-01-09 08:14:46 +08:00
bbfb8fd41c [branch-2.1] Fix local data dir metric missing (#46200) (#46603)
pick #46200
2025-01-09 00:03:13 +08:00
4a08d81acf [fix](set operator) Fix invalid access in set operator (#46528) (#46613)
Introduced by #45207.
For a query plan,
```
set sink operator -- -|
                                    |-> set source operator
set probe operator --|
```
If `set source operators` are finished (due to limit reached), all
upstream operators could be finished and waken up at the same time.
However, some states are initialized in `set sink operator`. So if `set
probe operator` executes before `set sink operator` initialization, it
will incur an invalid address access.

*** Query id: cebb723bbda64249-9ab8c9e7aa72c540 *** *** is nereids: 1
***
*** tablet id: 0 ***
*** Aborted at 1736092087 (unix time) try "date -d @1736092087" if you
are using GNU date ***
*** Current BE git commitID: 26d68d778a ***
*** SIGSEGV address not mapped to object (@0xf8) received by PID 23579
(TID 26524 OR 0x7f84d4240640) from PID 248; stack trace: *** 0#
doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at

/home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2#
JVM_handle_linux_signal in
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F88F4063520 in /lib/x86_64-linux-gnu/libc.so.6
4#

doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&)
at

/home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:183
5# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*,
doris::vectorized::Block*, bool) at

/home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:99
6# doris::pipeline::PipelineTask::execute(bool*) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:383
7# doris::pipeline::TaskScheduler::_do_work(int) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/task_scheduler.cpp:138
8# doris::ThreadPool::dispatch_thread() in
/mnt/hdd01/ci/master-deploy/be/lib/doris_be
9# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 10#
start_thread at ./nptl/pthread_create.c:442
11# 0x00007F88F4147850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
2025-01-08 21:09:33 +08:00
4d0037a928 branch-2.1: [fix](ES catalog)Fix query long value exception with doc_value #46554 (#46581)
Cherry-picked from #46554

Co-authored-by: qiye <luen@selectdb.com>
2025-01-08 15:26:58 +08:00
e90480037b [fix](index compaction)Fix construct index compaction columns core when reader close error (#46508)
fix #46507
2025-01-08 10:39:37 +08:00
f360ee23df [branch-2.1](metrics) Fix scan task related metrics (#46509)
### What problem does this PR solve?

Remove incorrect metrics
2025-01-08 10:10:03 +08:00
8aa2fafff4 [bugfix](asan core) should use weak ptr in rpc context to release rf context during query context deconstructed (#46565)
cherry-pick  #38653

Issue Number: close #xxx
F20240731 10:38:53.012670 20850 mem_tracker_limiter.cpp:135] mem tracker
label: Query#Id=7539c7d0087442b7-a1cda6392053548a, consumption: 52, peak
consumption: 1155332, mem tracker not equal to 0 when mem tracker
destruct, this usually means that memory tracking is inaccurate and
SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not
used correctly. 1. For query and load, memory leaks may have occurred,
it is expected that the query mem tracker will be bound to the thread
context using SCOPED_ATTACH_TASK and
SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER before all memory alloc and
free. 2. If a memory alloc is recorded by this tracker, it is expected
that be recorded in this tracker when memory is freed. 3. Merge the
remaining memory tracking value by this tracker into Orphan, if you
observe that Orphan is not equal to 0 in the mem tracker web or log,
this indicates that there may be a memory leak. 4. If you need to
transfer memory tracking value between two trackers, can use
transfer_to..[Address Sanitizer]:
0x606002867d80, size 52, strack trace:
0# Allocator::alloc_impl(unsigned long, unsigned long) 1# void*
phmap::priv::Allocate<4ul, doris::vectorized::Allocator_
>(doris::vectorized::Allocator_*, unsigned long)
2# phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::initialize_slots(unsigned long) 3#
phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::resize(unsigned long) 4#
phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::prepare_insert(unsigned long) 5#
std::pair phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::find_or_prepare_insert(int const&,
unsigned long)
6# std::pair, phmap::Hash, phmap::EqualTo, doris::vectorized::Allocator_
>::iterator, bool> phmap::priv::raw_hash_set, phmap::Hash,
phmap::EqualTo, doris::vectorized::Allocator_
>::emplace_decomposable(int const&, unsigned long, int const&)
7# std::pair, phmap::Hash, phmap::EqualTo, doris::vectorized::Allocator_
>::iterator, bool> phmap::priv::raw_hash_set, phmap::Hash,
phmap::EqualTo, doris::vectorized::Allocator_
>::EmplaceDecomposable::operator()(int const&, int const&) const
8# doris::HybridSet<(doris::PrimitiveType)5, doris::DynamicContainer,
doris::vectorized::ColumnVector >::insert(void const*) 9#
doris::HybridSetBase::insert(doris::HybridSetBase*) 10#
doris::RuntimePredicateWrapper::merge(doris::RuntimePredicateWrapper
const*)
11# doris::IRuntimeFilter::merge_from(doris::RuntimePredicateWrapper
const*)
12#

doris::RuntimeFilterMergeControllerEntity::merge(doris::PMergeFilterRequest
const*, butil::IOBufAsZeroCopyInputStream*)
13# doris::FragmentMgr::merge_filter(doris::PMergeFilterRequest const*,
butil::IOBufAsZeroCopyInputStream*)
14# std::_Function_handler::_M_invoke(std::_Any_data const&) 15#
doris::WorkThreadPool::work_thread(int)
16# execute_native_thread_routine
17# ?
18# ?
2025-01-08 09:30:29 +08:00
Pxl
2f98a6216e [Bug](join) fix columnstr64's offset overflow on serialize_value_into_arena #46461 (#46462)
pick from #46461
2025-01-07 19:59:35 +08:00
aa47a35384 [fix](mem) heap-buffer-overflow for function convert_to (#46405) (#46502)
pick #46405 to branch-2.1
2025-01-07 13:46:32 +08:00
03a3f37cc4 [fix](index meta) make has_inverted_index function more robust #46364 (#46428)
cherry pick from #46364
2025-01-07 09:36:22 +08:00
1405c48a77 [opt](coordinator) optimize parallel degree of shuffle when use nereids (#44754) (#46397)
cherry pick from #44754
2025-01-04 19:04:17 +08:00
db224ba15f [fix](variant) fix schema change for variant from not null to null (#46403)
cherry-pick from #46279
2025-01-04 09:00:43 +08:00