Commit Graph

21238 Commits

Author SHA1 Message Date
afeff5daee branch-2.1: [fix](nereids) fix bug in PhysicalTopN.equals() #46547 (#46633)
Cherry-picked from #46547

Co-authored-by: minghong <zhouminghong@selectdb.com>
2025-01-09 10:28:40 +08:00
52fd9349f0 [regression-test](case) Reduce test_load_to_single_tablet.groovy data size to prevent failure (#46655) 2025-01-09 08:18:18 +08:00
84ea1ee834 branch-2.1: [fix](suites) Fix atomic restore alter suite with master_sql #46550 (#46653)
Cherry-picked from #46550

Co-authored-by: walter <maochuan@selectdb.com>
2025-01-09 08:17:39 +08:00
f744b51c8c branch-2.1: [chore](regression) modify cos bucket of paimon case #46592 (#46642)
Cherry-picked from #46592

Co-authored-by: zgxme <zhenggaoxiong@selectdb.com>
2025-01-09 08:17:23 +08:00
b23877fe65 [regression-test](case) add info to debug test_workload_sched_policy.… (#46651) 2025-01-09 08:16:59 +08:00
ca5a4f6d35 [Fix](Variant) variant should not implicit be short key column when create mv (#46539)
cherry-pick from #46444
2025-01-09 08:15:22 +08:00
0ecb84c638 branch-2.1: [Fix](partial update) Fix incorrect result when partial update include delete sign columns #46194 (#46543)
pick https://github.com/apache/doris/pull/46194
2025-01-09 08:14:46 +08:00
01469e11cf [fix](regression-test) fix test_routine_load_eof case unstable (#46644)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

Co-authored-by: yiguolei <guolei@selectdb.com>
2025-01-09 08:13:36 +08:00
bbfb8fd41c [branch-2.1] Fix local data dir metric missing (#46200) (#46603)
pick #46200
2025-01-09 00:03:13 +08:00
ebb425dd84 branch-2.1: [fix](planner) fix core when select and filter by slot in old planner #46541 (#46638)
Cherry-picked from #46541

Co-authored-by: Lijia Liu <liutang123@yeah.net>
Co-authored-by: liutang123 <liulijia@gmail.com>
2025-01-08 22:32:35 +08:00
b99ef07a3b branch-2.1: [fix](Nereids) fix log and power #46077 (#46635)
Cherry-picked from #46077

Co-authored-by: LiBinfeng <libinfeng@selectdb.com>
2025-01-08 22:31:22 +08:00
f2419c49a7 branch-2.1: [opt](jni) Modify Exception Cause Formatting in JniUtil for Better Readability #46437 (#46602)
Cherry-picked from #46437

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-01-08 22:30:19 +08:00
ba091ae4bd [fix](ci) tmp exclude block case test_cumu_compaction_with_delete (#46649) 2025-01-08 22:00:26 +08:00
6a68b09100 branch-2.1: [fix](cache) fix mtmv cache can not use test #46537 (#46607)
Cherry-picked from #46537

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
2025-01-08 21:09:53 +08:00
4a08d81acf [fix](set operator) Fix invalid access in set operator (#46528) (#46613)
Introduced by #45207.
For a query plan,
```
set sink operator -- -|
                                    |-> set source operator
set probe operator --|
```
If `set source operators` are finished (due to limit reached), all
upstream operators could be finished and waken up at the same time.
However, some states are initialized in `set sink operator`. So if `set
probe operator` executes before `set sink operator` initialization, it
will incur an invalid address access.

*** Query id: cebb723bbda64249-9ab8c9e7aa72c540 *** *** is nereids: 1
***
*** tablet id: 0 ***
*** Aborted at 1736092087 (unix time) try "date -d @1736092087" if you
are using GNU date ***
*** Current BE git commitID: 26d68d778a ***
*** SIGSEGV address not mapped to object (@0xf8) received by PID 23579
(TID 26524 OR 0x7f84d4240640) from PID 248; stack trace: *** 0#
doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at

/home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2#
JVM_handle_linux_signal in
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F88F4063520 in /lib/x86_64-linux-gnu/libc.so.6
4#

doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&)
at

/home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:183
5# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*,
doris::vectorized::Block*, bool) at

/home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:99
6# doris::pipeline::PipelineTask::execute(bool*) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:383
7# doris::pipeline::TaskScheduler::_do_work(int) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/task_scheduler.cpp:138
8# doris::ThreadPool::dispatch_thread() in
/mnt/hdd01/ci/master-deploy/be/lib/doris_be
9# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 10#
start_thread at ./nptl/pthread_create.c:442
11# 0x00007F88F4147850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
2025-01-08 21:09:33 +08:00
3bc70876c4 branch-2.1: [fix](test) Optimize the health check after oceanbase docker starts #46434 (#46599)
Cherry-picked from #46434

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-01-08 20:29:40 +08:00
c805e69171 branch-2.1: [regression-test](fix) fix test_bloom_filter_hit.groovy case bug #46438 (#46617)
Cherry-picked from #46438

Co-authored-by: shuke <shuke@selectdb.com>
2025-01-08 15:51:23 +08:00
ddeec1426d [fix](test)Fix unstable index_change_with_cumu case (#46527)(#38733) (#46529)
bp: #46527, #38733
2025-01-08 15:41:43 +08:00
4d0037a928 branch-2.1: [fix](ES catalog)Fix query long value exception with doc_value #46554 (#46581)
Cherry-picked from #46554

Co-authored-by: qiye <luen@selectdb.com>
2025-01-08 15:26:58 +08:00
2687dedabf [fix](nereids) fix sql cache bug and some tests (#46443) (#46505)
cherry pick from #46443
2025-01-08 14:45:06 +08:00
f6ea71ab61 branch-2.1: [chore](MTMV) catch all exception when get mtmvs to avoid plan failed #46474 (#46536)
Cherry-picked from #46474

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-01-08 13:56:18 +08:00
eda8a26ec8 [ci](fix) tmp exclude case test_routine_load_eof (#46586) 2025-01-08 11:12:53 +08:00
5d2930e783 [fix](shellcheck) fix hive-metastore and enable shellcheck in docker (#46496) (#46574)
cherry-pick (#46496)

Co-authored-by: Socrates <suyiteng@selectdb.com>
2025-01-08 11:10:34 +08:00
e90480037b [fix](index compaction)Fix construct index compaction columns core when reader close error (#46508)
fix #46507
2025-01-08 10:39:37 +08:00
f360ee23df [branch-2.1](metrics) Fix scan task related metrics (#46509)
### What problem does this PR solve?

Remove incorrect metrics
2025-01-08 10:10:03 +08:00
4b3422fc8e branch-2.1: [fix](bf index) add ngram bf index validation in nereids index definition check #45780 (#46216)
cherry pick from #45780
2025-01-08 10:04:21 +08:00
8aa2fafff4 [bugfix](asan core) should use weak ptr in rpc context to release rf context during query context deconstructed (#46565)
cherry-pick  #38653

Issue Number: close #xxx
F20240731 10:38:53.012670 20850 mem_tracker_limiter.cpp:135] mem tracker
label: Query#Id=7539c7d0087442b7-a1cda6392053548a, consumption: 52, peak
consumption: 1155332, mem tracker not equal to 0 when mem tracker
destruct, this usually means that memory tracking is inaccurate and
SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not
used correctly. 1. For query and load, memory leaks may have occurred,
it is expected that the query mem tracker will be bound to the thread
context using SCOPED_ATTACH_TASK and
SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER before all memory alloc and
free. 2. If a memory alloc is recorded by this tracker, it is expected
that be recorded in this tracker when memory is freed. 3. Merge the
remaining memory tracking value by this tracker into Orphan, if you
observe that Orphan is not equal to 0 in the mem tracker web or log,
this indicates that there may be a memory leak. 4. If you need to
transfer memory tracking value between two trackers, can use
transfer_to..[Address Sanitizer]:
0x606002867d80, size 52, strack trace:
0# Allocator::alloc_impl(unsigned long, unsigned long) 1# void*
phmap::priv::Allocate<4ul, doris::vectorized::Allocator_
>(doris::vectorized::Allocator_*, unsigned long)
2# phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::initialize_slots(unsigned long) 3#
phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::resize(unsigned long) 4#
phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::prepare_insert(unsigned long) 5#
std::pair phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::find_or_prepare_insert(int const&,
unsigned long)
6# std::pair, phmap::Hash, phmap::EqualTo, doris::vectorized::Allocator_
>::iterator, bool> phmap::priv::raw_hash_set, phmap::Hash,
phmap::EqualTo, doris::vectorized::Allocator_
>::emplace_decomposable(int const&, unsigned long, int const&)
7# std::pair, phmap::Hash, phmap::EqualTo, doris::vectorized::Allocator_
>::iterator, bool> phmap::priv::raw_hash_set, phmap::Hash,
phmap::EqualTo, doris::vectorized::Allocator_
>::EmplaceDecomposable::operator()(int const&, int const&) const
8# doris::HybridSet<(doris::PrimitiveType)5, doris::DynamicContainer,
doris::vectorized::ColumnVector >::insert(void const*) 9#
doris::HybridSetBase::insert(doris::HybridSetBase*) 10#
doris::RuntimePredicateWrapper::merge(doris::RuntimePredicateWrapper
const*)
11# doris::IRuntimeFilter::merge_from(doris::RuntimePredicateWrapper
const*)
12#

doris::RuntimeFilterMergeControllerEntity::merge(doris::PMergeFilterRequest
const*, butil::IOBufAsZeroCopyInputStream*)
13# doris::FragmentMgr::merge_filter(doris::PMergeFilterRequest const*,
butil::IOBufAsZeroCopyInputStream*)
14# std::_Function_handler::_M_invoke(std::_Any_data const&) 15#
doris::WorkThreadPool::work_thread(int)
16# execute_native_thread_routine
17# ?
18# ?
2025-01-08 09:30:29 +08:00
Pxl
2f98a6216e [Bug](join) fix columnstr64's offset overflow on serialize_value_into_arena #46461 (#46462)
pick from #46461
2025-01-07 19:59:35 +08:00
6bf31a2a0d branch-2.1: [fix](variable) variable version not persist in image #46463 (#46499)
Cherry-picked from #46463

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-01-07 16:41:49 +08:00
45eaa3862d [regression-test](fix) fix test_audit_log_behavior.groovy bug (#46100) (#46524)
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
In this case, select stmt from __internal_schema.audit_log where stmt
like '%3F6B9A_1%' order by time asc limit 1
will also be inserted into audit log table, and be selected, which will
cause assert fail in test case, which want a insert stmt.
2025-01-07 14:38:25 +08:00
c9ab3b9e3a [regression-test](fix) ignore E-2000 in compaction cases (#46522)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx
#46436
2025-01-07 14:26:54 +08:00
aa47a35384 [fix](mem) heap-buffer-overflow for function convert_to (#46405) (#46502)
pick #46405 to branch-2.1
2025-01-07 13:46:32 +08:00
abbf75e9fe branch-2.1: [fix](Nereids) insert into with cte cannot be explained (#46384) (#46501)
pick from master #46384
2025-01-07 13:44:32 +08:00
52455ed6d5 branch-2.1: [opt](cache) enhance cache key computation by removing comments and trimming SQL input #46099 (#46472)
Cherry-picked from #46099

Co-authored-by: York Cao <52438394+Baymine@users.noreply.github.com>
2025-01-07 13:39:48 +08:00
5962dceddb branch-2.1: [fix](hms)fix hive catalog follower not get event. #46485 (#46490)
Cherry-picked from #46485

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-01-07 12:19:06 +08:00
8f4b76a5f3 [fix](regression-test) adjust hudi cases (#46473) (#46488)
cherry-pick #46473

Co-authored-by: Socrates <suyiteng@selectdb.com>
2025-01-07 12:11:34 +08:00
5e9847a778 [regression-test](fix) fix global var bug in compaction/test_cumu_compaction_with_delete.groovy (#46514) 2025-01-07 11:41:05 +08:00
03a3f37cc4 [fix](index meta) make has_inverted_index function more robust #46364 (#46428)
cherry pick from #46364
2025-01-07 09:36:22 +08:00
ccb6c2610a branch-2.1: [fix](nereids)fix bug of PushDownFilterThroughSetOperation (#46370) (#46456)
pick from master #46370

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2025-01-06 21:34:40 +08:00
53889cbbb6 branch-2.1: [enhance](auth)Reduce the scope of permission locks #46115 (#46451)
Cherry-picked from #46115

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-01-06 18:51:09 +08:00
4b69673e4e [fix](fe.conf) use g1 by default for jdk17 (#46395) 2025-01-06 17:52:35 +08:00
dda058c433 branch-2.1: [fix](test)fix unstable case correctness_p0/test_mv_case/test_mv_case #46101 (#46431)
Cherry-picked from #46101

Co-authored-by: starocean999 <lichi@selectdb.com>
2025-01-06 17:42:40 +08:00
6958dec242 branch-2.1: [fix](regression)Fix schema change statistics case. Neredis not allow change MV schema. #46433 (#46448)
Cherry-picked from #46433

Co-authored-by: James <lijibing@selectdb.com>
2025-01-06 16:50:52 +08:00
7b8a93879c [test](mtmv)Test and check the differences between MTMV and OLAP (#44793)
pick from #36520 
pick from #44417 

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-01-06 09:57:59 +08:00
12350ca867 branch-2.1: [Improve](mtmv) skip the generation of invalid task for refresh mtmv #46280 (#46392)
Cherry-picked from #46280

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: garenshi <garenshi@tencent.com>
2025-01-05 08:59:11 +08:00
a4c328c4cd branch-2.1: [fix](compile) fix fe compile failed by code too large #46408 (#46420)
Cherry-picked from #46408

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
2025-01-05 05:07:49 +08:00
3d6ad47d08 branch-2.1: [fix](test) fix unstable lower case test #46410 (#46416)
Cherry-picked from #46410

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-01-04 22:28:29 +08:00
eb867f593a branch-2.1: [opt](iceberg) opt the error msg when create db which already exists #46334 (#46414)
Cherry-picked from #46334

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-01-04 19:05:52 +08:00
1405c48a77 [opt](coordinator) optimize parallel degree of shuffle when use nereids (#44754) (#46397)
cherry pick from #44754
2025-01-04 19:04:17 +08:00
1b1d7e2f29 branch-2.1: [fix](test) fix test_autoinc_broker_load.groovy #46320 (#46338)
Cherry-picked from #46320

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-01-04 14:17:31 +08:00