Commit Graph

8558 Commits

Author SHA1 Message Date
c3707dbdee branch-2.1: [Enhancement](function) Print exact column name when execute non_nullable failed #44679 (#44742)
Cherry-picked from #44679

Co-authored-by: zclllhhjj <zhaochangle@selectdb.com>
2024-11-29 20:52:19 +08:00
412cec5bad branch-2.1: [fix](clone) Fix wrong clone file path #44746 (#44773)
Cherry-picked from #44746

Co-authored-by: walter <maochuan@selectdb.com>
2024-11-29 14:09:16 +08:00
82c7a9d15a [Fix](Variant) create table should not automatically add variant to key (#44736)
#36609
2024-11-29 09:34:43 +08:00
Pxl
bfd51f07e6 revert #44408 on branch-2.1 (#44757) 2024-11-29 08:28:46 +08:00
Pxl
846fe83152 [Chore](runtime-filter) add rpc error msg to RuntimeFilterContext (#43517) (#44622) (#44719)
pick from #43517
2024-11-28 16:46:27 +08:00
e7520ae6cf branch-2.1: [fix](hyperscan) Fix hyper scan fall back to re2 #44547 (#44653)
Cherry-picked from #44547

Co-authored-by: zhiqiang <hezhiqiang@selectdb.com>
2024-11-28 16:00:43 +08:00
Pxl
ddaaac5b44 [Bug](runtime-filter) pick #44408 #43078 (#44686)
pick #44408 #43078
2024-11-28 12:23:38 +08:00
88cfaedb16 [opt](paimon)Optimize the storage location of the serialized paimon table for 2.1 (#44274) (#44660)
bp: #44274
2024-11-27 20:35:35 +08:00
66cb95ffed branch-2.1: [fix](inverted index) Content Check for Tokenize Function Parser #44465 (#44555)
Cherry-picked from #44465

Co-authored-by: zzzxl <yangsiyu@selectdb.com>
2024-11-27 11:53:36 +08:00
Pxl
7464f461bc [Bug](runtime-filter) avoid ignore rf multiple times (#44408) (#44625)
avoid ignore rf multiple times

When an instance is wake_up_by_downstream, it will destroy the logic of
`first closing the instance that should build hash table and then
closing other instances`, instance that should build hash table during
the process of inserting data into rf, this instance may find that rf is
ignored, causing dcheck to fail.

```cpp
 F20241121 12:01:02.245405 9832 runtime_filter.cpp:395] Check failed: !is_ignored()

3# raise at ../sysdeps/posix/raise.c:27
4# abort at ./stdlib/abort.c:81
5# 0x00005603CFDF778D in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
6# 0x00005603CFDE9DCA in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
7# google::LogMessage::SendToLog() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
8# google::LogMessage::Flush() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
9# google::LogMessageFatal::~LogMessageFatal() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
10# doris::RuntimePredicateWrapper::insert_fixed_len(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn> const&, unsigned long) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/exprs/runtime_filter.cpp:395
11# doris::RuntimePredicateWrapper::insert_batch(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn> const&, unsigned long) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/exprs/runtime_filter.cpp:431
12# doris::IRuntimeFilter::insert_batch(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>, unsigned long) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/exprs/runtime_filter.cpp:992
13# doris::VRuntimeFilterSlots::insert(doris::vectorized::Block const*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/exprs/runtime_filter_slots.h:146
14# doris::pipeline::HashJoinBuildSinkLocalState::close(doris::RuntimeState*, doris::Status) in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
15# doris::pipeline::DataSinkOperatorXBase::close(doris::RuntimeState*, doris::Status) in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be
16# doris::pipeline::PipelineTask::close(doris::Status) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:487
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2024-11-27 10:51:02 +08:00
43ebfa9031 [branch-2.1] pick some workload group pr (#44634)
pick #42053 #43942
2024-11-27 09:42:41 +08:00
f88a8169dc [opt](Variant) avoid unnecessary mem for variant extracted columns (#… (#44585)
cherry-pick from #43567
2024-11-26 19:30:06 +08:00
071a2106a3 branch-2.1: [opt](brpc) Make the timeout of brpc checking configurable #44481 (#44602)
Cherry-picked from #44481

Co-authored-by: Jerry Hu <hushenggang@selectdb.com>
2024-11-26 19:26:39 +08:00
9bc547578c [opt](scan) Release instances of Segment to avoid consuming a large amount of memory in ParallelScannerBuilder (#44189) (#44608) 2024-11-26 19:25:58 +08:00
bdca9cce3e [branch-2.1]add internal workload group (#42006) (#44592)
Add an internal workload group when Doris started, currently it mainly
used to manage compaction workload cpu usage.
pick #42006
2024-11-26 17:09:55 +08:00
550aa9c3ae [cherry-pick](branch-2.1) Pick "[Fix](full compaction) Full compaction should not do ordered data compaction (#44359)" (#44529) 2024-11-26 12:21:30 +08:00
cf2ee23e68 Revert "[fix] (compaction) fix compaction score in time series policy (#40242)" (#44527)
revert #40242
2024-11-26 12:18:59 +08:00
5eb433e7aa [branch-2.1] pick some pr from master (#44556)
pick #40468 #40535  #40763 #40855
2024-11-25 20:43:07 +08:00
a233343fe3 branch-2.1: [fix](jvm)fix jvm metrics memory leak. #44311 (#44399)
Cherry-picked from #44311

Co-authored-by: daidai <changyuwei@selectdb.com>
2024-11-23 03:30:28 +08:00
6678088767 [cherry-pick](branch-21) reuse the join block to reduce malloc memory (#43738) (#44349)
cherry-pick from master https://github.com/apache/doris/pull/43738/
2024-11-23 03:29:52 +08:00
92a0919e9c branch-2.1: [Bug](function)fix json_object function check null nums error as return bool #44321 (#44381)
Cherry-picked from #44321

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
2024-11-22 23:51:33 +08:00
d21940ee1c branch-2.1: [fix](move-memtable) immediately return error when close wait failed #44344 (#44387)
Cherry-picked from #44344

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
2024-11-22 22:56:51 +08:00
dceaf97381 branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate #43255 (#44438)
Cherry-picked from #43255

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-11-22 22:52:53 +08:00
702abbff0f [Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. (#42004) (#44239)
bp #42004

Co-authored-by: kaka11chen <kaka11.chen@gmail.com>
2024-11-22 11:01:41 +08:00
346b89e683 [improve](routine load) adjust default values to make routine load more convenient to use (#42491) (#44377)
pick (#42491)

For a routine load job, it will be divided into many tasks, each of
which is a transaction. Currently, the default time
consumed(max_batch_interval) is 10 seconds. The benefits of increasing
this value are:
1. Larger batch consumption can lead to better performance.
2. Reducing the number of transactions can alleviate the pressure of
compaction and the conflicts of concurrent transaction submissions.

related doc: https://github.com/apache/doris-website/pull/1236/files
2024-11-21 23:05:11 +08:00
9664b50eb6 [improve](load) do not block delta writer if memtable memory is low (#42649) (#44305)
backport #42649
2024-11-21 11:17:35 +08:00
fb163b55c2 branch-2.1: [Fix](merge-on-write) Fix MergeIndexDeleteBitmapCalculator::calculate_one() coredump #44284 (#44330)
Cherry-picked from #44284

Co-authored-by: bobhan1 <baohan@selectdb.com>
2024-11-20 21:07:43 +08:00
dc67086d97 [fix](scan) Avoid memory allocated by buffered_reader from being traced (#41921) (#44253)
Use OwnedSlice to replace `char*` in BufferedReader

## Proposed changes

pick #41921
2024-11-20 10:37:06 +08:00
610054c77b [cherry-pick](branch-21) fix exchange of tablet shuffle send block error (#44102) (#44230)
cherry-pick from master (#44102)
2024-11-19 17:31:06 +08:00
433c1bc9ff [cherry-pick](branch-21) replace the LOG(FATAL) to throw Exception in query execute layer (#38144) (#44183)
cherry-pick from master https://github.com/apache/doris/pull/38144
2024-11-19 17:28:20 +08:00
c9801f7a38 branch-2.1: [Bug](function) fix cut_ipv6 function error about modify the input column data #43921 (#44180)
Cherry-picked from #43921

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
2024-11-19 17:27:29 +08:00
83208ee1a8 [pick](branch-2.1) pick #43960 #43929 #44177 (#44240)
pick #43960 #43929 #44177
2024-11-19 17:25:16 +08:00
eeafe45c0c [fix](brpc) coredump caused by brpc checking (#44047) (#44188)
pick #44047
```
/root/doris/be/src/runtime/fragment_mgr.cpp:1064:20: runtime error: member call on null pointer of type 'doris::PBackendService_Stub'

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/runtime/fragment_mgr.cpp:1064:20 in
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1731663847 (unix time) try "date -d @1731663847" if you are using GNU date ***
*** Current BE git commitID: b663df0e50 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 17169 (TID 17463 OR 0x7f746d21a700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo_t*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
3# 0x00007F7601263090 in /lib/x86_64-linux-gnu/libc.so.6
4# doris::FragmentMgr::_check_brpc_available(std::shared_ptr<doris::PBackendService_Stub> const&, doris::FragmentMgr::BrpcItem const&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
5# doris::FragmentMgr::cancel_worker() at /root/doris/be/src/runtime/fragment_mgr.cpp:1022
6# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:499
7# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
8# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2024-11-19 14:56:46 +08:00
b4e136bfcc [performance](move-memtable) async close tablet streams (#41156 & #43813) (#44128)
backport #41156 and #43813
2024-11-19 14:14:53 +08:00
fa43cc0a90 [improve](http) Save the requested url in http execution error #43855 (#44106)
cherry pick from #43855
2024-11-18 15:06:02 +08:00
445e196041 [improve](load) pass cancel reason to tablet writer when cancelled (#43388) (#44131)
backport #43388
2024-11-18 14:11:38 +08:00
ea61206233 [pick](branch-2.1) pick #43281 (#44020)
pick #43281
2024-11-16 21:53:21 +08:00
261c65f72d [fix](pipeline) only sub_running_sink_operators in close #43500 (#43726)
https://github.com/apache/doris/pull/43500
### What problem does this PR solve?
Previously, sub_running_sink_operators was called only when encountering
EOS during sink
or when all sources were closed. However, this approach has issues, as
it’s possible
for the user to manually cancel, in which case there may be no EOS and
the sources may
not be closed. This would prevent running_sink_operators from reaching
zero, leading to errors.
```
PipelineTask[this = 0x7fc369fe9600, id = 0, open = true, eos = false, finish = false, dry run = false, elapse time = 26361.740784032s], block dependency = NULL, is running = true
operators: 
LOCAL_EXCHANGE_OPERATOR (LOCAL_MERGE_SORT): id=-5, parallel_tasks=4, _channel_id: 0, _num_partitions: 4, _num_senders: 4, _num_sources: 4, _running_sink_operators: 1, _running_source_operators: 1, mem_usage: 0, data queue info: Data Queue 0: [size approx = 0, eos = false], MemTrackers: 0: 0, 1: 34537472, 2: 5701632, 3: 0, 
  DATA_STREAM_SINK_OPERATOR: id=6, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false
0.   this=0x7fc376438f10, LOCAL_MERGE_EXCHANGE_OPERATOR_DEPENDENCY: id=-5, block task = 0, ready=true, _always_ready=true
0.   this=0x7fc3764bc110, LOCAL_MERGE_EXCHANGE_OPERATOR_DEPENDENCY: id=-5, block task = 0, ready=true, _always_ready=true
0.   this=0x7fc3764bc310, LOCAL_MERGE_EXCHANGE_OPERATOR_DEPENDENCY: id=-5, block task = 0, ready=true, _always_ready=true
0.   this=0x7fc3764bc510, LOCAL_MERGE_EXCHANGE_OPERATOR_DEPENDENCY: id=-5, block task = 0, ready=true, _always_ready=true
```
- [x] Confirm test cases
- [x] Confirm document
- [x] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2024-11-16 20:58:47 +08:00
10009dc062 [fix](variant) fix index in variant (#43375) (#43971)
Problem Summary:

1. Fixing error checks,
`InvertedIndexColumnWriter::check_support_inverted_index`, It is not
appropriate to determine support for other indexes by checking the
inverted index.
2. Fix `TableSchema::update_index()`

pick from master #43375
2024-11-16 16:29:00 +08:00
d1cc68a26a branch-2.1: [Fix](auto-increment) Fix duplicate auto-increment column value problem #43774 (#43984)
Cherry-picked from #43774

Co-authored-by: bobhan1 <baohan@selectdb.com>
2024-11-16 16:17:24 +08:00
48e33bfb2a branch-2.1: [fix](hive)Fixed the issue of reading hive table with empty lzo files #43979 (#44063)
Cherry-picked from #43979

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
2024-11-16 16:14:50 +08:00
b3022df716 [fix](inverted index) base compaction failed after restore indexes (#43962)
### What problem does this PR solve?
**Problem:**
after restore from other cluster, then rowsets got different index_id,
and make index compaction in base compaction always failed.

**Fix:**
On master branch, this pr: https://github.com/apache/doris/pull/41625
already fix it.
Here pick it to branch-2.1
2024-11-16 16:01:13 +08:00
Pxl
21b3e4bbf9 [Bug](runtime-filter) fix core dump on rf between varchar and char #43758 (#43934)
#43758
2024-11-16 15:59:19 +08:00
edd9015de3 [branch-2.1](function) fix error result in auto partition name (#41130) (#43977)
pick https://github.com/apache/doris/pull/41130
https://github.com/apache/doris/pull/41372

---------

Co-authored-by: zhaochangle <zhaochangle@selectdb.com>
2024-11-15 19:11:42 +08:00
a519702fac branch-2.1: [Bug](bitmap-filter) fix wrong type cast on BitmapFilterColumnPredicate::evaluate #43877 (#43886)
Cherry-picked from #43877

Co-authored-by: Pxl <xl@selectdb.com>
2024-11-15 00:02:51 +08:00
7fc78e3f87 [opt](brpc) check and remove unavailable brpc stubs (#43212) (#43859) 2024-11-14 19:52:06 +08:00
26fef2a02a branch-2.1: [Fix](UT) Fix status UT error introduced by #43731 #43922 (#43927)
Cherry-picked from #43922

Co-authored-by: abmdocrt <lianyukang@selectdb.com>
2024-11-14 19:23:37 +08:00
39b9b81d42 branch-2.1:[fix](build) Fix Mac compilation error caused by namespace conflict in find_symbols.h (#43868) 2024-11-14 10:10:12 +08:00
47f842b4d6 branch-2.1: [minor](rpc) Check client before RPC (#43818)
Cherry-picked from #43626

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
2024-11-13 19:31:55 +08:00
d4712aed1a branch-2.1: [fix](string64) fix coredump caused by ColumnArray<ColumnStr<uint64_t>>::insert_indices_from (#43862)
Cherry-picked from #43624

Co-authored-by: TengJianPing <tengjianping@selectdb.com>
2024-11-13 19:31:11 +08:00