Commit Graph

8590 Commits

Author SHA1 Message Date
Pxl
98c782bedb [Chore](case) pick FragmentMgr::send_filter_size.return_eof/RuntimeFilterProducer::send_size.rpc_fail to 2.1 (#48817)
part of https://github.com/apache/doris/pull/48225
2025-03-08 16:21:22 +08:00
48e3a73118 [fix](cancel) Fix cancel failure (#48751)
### What problem does this PR solve?

If a query is canceled before fragment contexts are prepared, no
fragment context will be found. So we should set execution ready to
ensure tasks will not be blocked.
2025-03-07 11:10:58 +08:00
3f684f2899 branch-2.1:[fix] (inverted index) Fix UTF-8 4-byte truncation issue and add configuration to control correct term writing (#48657) (#48741)
Cherry-picked from #48657
2025-03-06 21:28:24 +08:00
7b2899a7ff [cherry-pick](jsonb) add a check for jsonb value to avoid invalid jsonb value write into segment file (#48729)
…ke select core (#48625)

fix invalid jsonb value write into segment file which make select core,
so we add a check for jsonb value when convert_to_olap which value will
be written into segment file
2025-03-06 15:50:35 +08:00
d61737f04b branch-2.1: [fix](function) fix the function elt #48701 (#48719)
Cherry-picked from #48701

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
2025-03-06 11:39:54 +08:00
c9a299e914 [fix](columns) fix bug found by UT and add regression test (#48554) (#48690)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR:  Pick #48554

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-03-06 09:33:33 +08:00
03514b476f Revert "branch-2.1: [fix](inverted index) fix wrong read data for primary key #47841 (#48207)" (#48684)
This reverts commit #48207 for branch2.1 and #47841 for master

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-03-05 18:51:54 +08:00
84c638ee68 [opt](Inverted index)Avoid repeated calculations of suffix paths (#48137) (#48155)
bp #48137
2025-03-05 17:56:16 +08:00
fa9c05a54a [fix](inverted index) Fix for Inaccurate match_phrase_prefix Cache in Query Processing (#48604)
https://github.com/apache/doris/pull/46310
2025-03-05 16:01:28 +08:00
1b33f8cfd6 branch-2.1: [fix](hudi) Set Spark Hudi JNI scanner as default (#48602) (#48606)
### What problem does this PR solve?
Related PR: #45041 

Problem Summary:
If we set hudi_jni_scanner to an incorrect value, jni_connector will be
null, causing a core dump.
So we set Spark Hudi JNI scanner as default and the hadoop hudi jni
reader will be supported in future
2025-03-05 14:02:46 +08:00
1b0bb4010b [Enhancement-2.1](log) Reduce INFO log size by changing some routine query log to VLOG (#48293)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

They are only useful when there's a query hanging in BE. When it occurs,
we can dynamically change the vlog level.

### Release note

Reduce query-side INFO log quantity

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [x] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-03-05 10:18:21 +08:00
be6210d742 branch-2.1: [improve](load) improve error message "unknown load_id" #47509 (#48639)
Cherry-picked from #47509

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
2025-03-05 10:11:24 +08:00
74b85d6bff branch-2.1-pick: [compaction](config) Add a config to control whether to prune rows with delete sign=1 in base compaction (#48241) (#48620)
pick https://github.com/apache/doris/pull/48241
2025-03-05 10:10:58 +08:00
fa4b901ec4 branch-2.1: [chore](http) add HttpClient::execute debug log #48595 (#48619)
Cherry-picked from #48595

Co-authored-by: walter <maochuan@selectdb.com>
2025-03-05 10:03:05 +08:00
621944d487 [InvertedIndex](Variant) supoort inverted index for array type in variant (#48594)
cherry-pick from #47688
2025-03-05 10:02:13 +08:00
08e7d920db branch-2.1: [fix](index build) Correct inverted index behavior after dynamically adding a column #48389 (#48546)
Cherry-picked from #48389

---------

Co-authored-by: airborne12 <jiangkai@selectdb.com>
2025-03-05 09:26:54 +08:00
Pxl
548c79f336 [Improvement](column) add santy check and add some fix for ColumnString #47964 (#48512)
pick part of #47964
2025-03-04 21:47:16 +08:00
1ccd879e3d branch-2.1: [fix](serde)fix arrow serde with no value into column #48053 (#48097)
Cherry-picked from #48053

Co-authored-by: amory <wangqiannan@selectdb.com>
2025-03-04 21:37:52 +08:00
c9381b0285 [fix](load) Fix import failure when the stream load parameter specifies Transfer-Encoding:chunked (#48196) (#48503)
pick from master  #48196
2025-03-04 10:12:54 +08:00
ddf63dc2f0 branch-2.1: [enhancement](threadpool) reduce thread pool for arrow flight and spill io threads #48530 (#48556)
Cherry-picked from #48530

Co-authored-by: yiguolei <guolei@selectdb.com>
2025-03-03 18:51:48 +08:00
cd3e1dce74 [feature](inverted index) Add profile statistics for each condition in inverted index filters (#48459)
https://github.com/apache/doris/pull/47504
2025-03-01 11:00:19 +08:00
c5e67bf82d branch-2.1: [Chore](client) Do not log in thrift exception when ADDRESS_SANITIZER is defined #48430 (#48455)
Cherry-picked from #48430

Co-authored-by: Pxl <xl@selectdb.com>
2025-02-28 15:12:12 +08:00
da90fba590 branch-2.1: [fix](client) Do not log in thrift exception when ADDRESS_SANITIZER is defined #48347 (#48404)
cherry pick from #48347
2025-02-27 15:52:39 +08:00
dae9d9d5e4 [cherry-pick](branch-2.1) Don't prematurely erase DeleteRows in reading iceberg table with position delete (#47977) (#48308)
### What problem does this PR solve?
Issue Number: close #41460
Problem Summary:
When reading the Iceberg table, previously read `DeleteRows` should not
be released immediately, as the Iceberg data file is split into multiple
`IcebergSplit`s for execution. These `IcebergSplit`s belong to the same
data file, meaning they share the same `DeleteRows`. Therefore,
`DeleteRows` in the `DeleteFile` should not be released prematurely.
Instead, they should be released when the shared_kv is reset, at which
point all `DeleteRows` will be freed along with the cached `DeleteFile`.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-02-27 15:44:40 +08:00
393cb00174 [fix](metrics) max_compaction_score metrics do not update while compaction_num_per_round > 1 (#48383)
refer to #46160
2025-02-27 11:07:15 +08:00
45ebb21cad branch-2.1-pick: [Opt](partial update) Add some cases for partial update (#47900) (#48272)
pick https://github.com/apache/doris/pull/47900
2025-02-26 22:04:57 +08:00
d0cb4f8864 branch-2.1: [fix](schema scan) Fix invalid pointer access #48370 (#48387)
Cherry-picked from #48370

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
2025-02-26 22:04:05 +08:00
4f3ccca9a2 branch-2.1: [fix](schema scan) Fix invalid pointer access #48313 (#48341)
Cherry-picked from #48313

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
2025-02-26 20:29:39 +08:00
b1ac39587b [fix](variant) fix variant used in order by back to legacy planner would meet core (#48332)
### What problem does this PR solve?
sql 
```
 SELECT    *,    ROW_NUMBER() OVER (     PARTITION BY orderid      ORDER BY v DESC   ) AS row_idfirst  FROM test_v_legacy order by orderid;
```
core:
```
F20250225 11:23:08.735848 1788347 column_object.h:439] should not call the method in column object
*** Check failure stack trace: ***
F20250225 11:23:08.736686 1788345 column_object.h:439] should not call the method in column object
*** Check failure stack trace: ***
    @     0x5555fa0821b6  google::LogMessage::SendToLog()
    @     0x5555fa0821b6  google::LogMessage::SendToLog()
    @     0x5555fa07ec00  google::LogMessage::Flush()
    @     0x5555fa07ec00  google::LogMessage::Flush()
    @     0x5555fa0829f9  google::LogMessageFatal::~LogMessageFatal()
    @     0x5555fa0829f9  google::LogMessageFatal::~LogMessageFatal()
    @     0x5555dd722b1c  doris::vectorized::ColumnObject::get_permutation()
    @     0x5555dd722b1c  doris::vectorized::ColumnObject::get_permutation()
    @     0x5555dd6b933c  doris::vectorized::ColumnNullable::get_permutation()
    @     0x5555dd6b933c  doris::vectorized::ColumnNullable::get_permutation()
    @     0x5555dde3997f  doris::vectorized::sort_block()
    @     0x5555dde3997f  doris::vectorized::sort_block()
    @     0x5555e2f13646  doris::vectorized::Sorter::partial_sort()
    @     0x5555e2f13646  doris::vectorized::Sorter::partial_sort()
    @     0x5555e2f1503b  doris::vectorized::FullSorter::_do_sort()
    @     0x5555e2f1503b  doris::vectorized::FullSorter::_do_sort()
    @     0x5555e2f15a2f  doris::vectorized::FullSorter::prepare_for_read()
    @     0x5555e2f15a2f  doris::vectorized::FullSorter::prepare_for_read()
    @     0x5555f97fb76a  doris::pipeline::SortSinkOperatorX::sink()
    @     0x5555f97fb76a  doris::pipeline::SortSinkOperatorX::sink()
    @     0x5555fa018798  doris::pipeline::PipelineXTask::execute()
    @     0x5555fa018798  doris::pipeline::PipelineXTask::execute()
    @     0x5555fa04efe5  doris::pipeline::TaskScheduler::_do_work()
    @     0x5555fa04efe5  doris::pipeline::TaskScheduler::_do_work()
    @     0x5555fa052f2b  doris::pipeline::TaskScheduler::start()::$_0::operator()()
    @     0x5555fa052f2b  doris::pipeline::TaskScheduler::start()::$_0::operator()()

```
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-02-26 20:27:43 +08:00
82fe8bc3d7 branch-2.1:[fix](libhdfs) fix the lifecycle issue of libhdfs config (#48352)
pick part of #47299
when calling `hdfsBuilderSetKerb5Conf`, the `value` string's lifecycle
must be with `hdfs_builder`.
2025-02-26 14:10:49 +08:00
69196d8212 [fix](jvm) the jvm opt should only be set once #48335 (#48336)
bp #48335
2025-02-26 09:52:58 +08:00
a2fe1bda7f branch-2.1: [fix](group commit) fix wal reader handle empty block #48290 (#48334)
Cherry-picked from #48290

Co-authored-by: meiyi <meiyi@selectdb.com>
2025-02-26 09:51:55 +08:00
babab64bbd branch-2.1: [fix](group commit) group commit failed if enable global enable_unique_key_partial_update (#48251)
pick https://github.com/apache/doris/pull/48120
2025-02-25 22:08:50 +08:00
4f8ff321ea branch-2.1: [fix](inverted index) fix wrong read data for primary key #47841 (#48207)
Cherry-picked from #47841

Co-authored-by: airborne12 <jiangkai@selectdb.com>
2025-02-25 21:15:03 +08:00
d6f4db4170 [fix](load) fix metrics memtable_flush_duration_us (#47607) (#48213)
backport #47607
2025-02-25 21:13:03 +08:00
Pxl
7a98ac434f [Chore](case) pick some broadcast join case from #48247 #47380 (#48310)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-02-25 20:32:12 +08:00
1aa57a3b13 branch-2.1: [fix](array index) Correct null bitmap writing for inverted index #47846 (#48214)
cherry pick from #47846 #48231
2025-02-25 20:31:18 +08:00
0d386f9268 branch-2.1: [fix](java) should use JAVA_OPTS_FOR_JDK_17 instead of JAVA_OPTS #48170 (#48282)
bp #48170
So that when user using jdk17 to run Doris 2.1, it will get the right
JAVA_OPTS
2025-02-25 18:23:18 +08:00
feb4b09cb3 [fix](hive)Spelling mistake of the word "failed" for 2.1 (#48193) 2025-02-22 23:18:21 +08:00
bcd01c5f24 branch-2.1-pick: [Fix](merge-on-write) should calculate delete bitmaps between segmens before skip if tablet is in NOT_READY state in flush phase (#48056) (#48089)
pick https://github.com/apache/doris/pull/48056
2025-02-21 19:53:17 +08:00
469bc77a42 branch-2.1: [fix](jdbc catalog) Change BE jdbc Driver loading to Java code (#48002)
cherry-pick from (#46912)
2025-02-21 17:39:26 +08:00
3fa4413e3b branch-2.1: [fix](restore) Lock tablet before modify segment files #45711 (#48047)
cherry pick from #45711
2025-02-21 08:33:10 +08:00
a6fe815f06 branch-2.1: [Bug](function) add index check on function like #47991 (#48019)
Cherry-picked from #47991

Co-authored-by: Pxl <xl@selectdb.com>
2025-02-19 15:22:51 +08:00
bc6af178b3 branch-2.1: [fix](parquet)Fix data column and null map column not equal when reading Parquet complex type cross-page data #47734 (#48039)
Cherry-picked from #47734

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-02-19 15:21:13 +08:00
7f3681b3d7 branch-2.1: [BugFix](Variant) limit cast elimination to one level cast #47778 (#47955)
Cherry-picked from #47778

Co-authored-by: lihangyu <lihangyu@selectdb.com>
2025-02-19 09:30:46 +08:00
9d5ec37309 branch-2.1: [fix](group commit) replay wal failed when enable global enable_memtable_on_sink_node #47968 (#48026)
Cherry-picked from #47968

Co-authored-by: meiyi <meiyi@selectdb.com>
2025-02-19 09:29:31 +08:00
c7ad5b6904 branch-2.1: [Bug](set) fix find null get wrong result on set operators #48001 (#48020)
Cherry-picked from #48001

Co-authored-by: Pxl <xl@selectdb.com>
2025-02-19 09:26:49 +08:00
c099ccdbd0 branch-2.1: [improve](load) print error string in local fs error messages #47918 (#48010)
Cherry-picked from #47918

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
2025-02-19 09:25:41 +08:00
e153103e4d branch-2.1: [improve](load) improve error message "close wait failed coz rpc error" #47518 (#47805)
Cherry-picked from #47518

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
2025-02-18 09:38:28 +08:00
f85176bee0 branch-2.1: [improve](move-memtable) improve error log and message for "not enough streams" #47470 (#47722)
Cherry-picked from #47470

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
2025-02-12 14:42:11 +08:00