Commit Graph

21621 Commits

Author SHA1 Message Date
6430ff365d [Bug](partition) should not do reset for the partition_sorts (#49148)
### What problem does this PR solve?
Problem Summary:

the partition_sorts is unique_ptr,could release after destroy.
and when some extreme case like cancel, if source reset the sorter
early, the sink operator will coredump still use it.
2025-03-18 18:03:29 +08:00
16e348b189 [fix](array/map) Fix BE crash in lambda functions (#49139) 2025-03-18 11:51:38 +08:00
bd5d0ca741 branch-2.1: [fix](Nereids) fold str_to_date to wrong result when parameter out of range #49033 (#49150) 2025-03-18 11:01:49 +08:00
68f50b569b branch-2.1: [fix](group commit)Fix replay wal fail problem on agg state type #49081 (#49143)
Cherry-picked from #49081

Co-authored-by: huanghaibin <huanghaibin@selectdb.com>
2025-03-18 10:57:20 +08:00
98a59f914a branch-2.1: [chore](binlog) GetMeta returns dropped partition/table/index commit seq #48852 (#48900)
Cherry-picked from #48852

Co-authored-by: walter <maochuan@selectdb.com>
2025-03-17 14:03:09 +08:00
cf88db4938 branch-2.1: [improvement](ccr) Add and adjust result for get_lag #48953 (#49055)
Cherry-picked from #48953

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2025-03-17 10:50:07 +08:00
e7d4dda3c8 branch-2.1: [Fix](function) Fix wrong FE fold constant implementation of function date_format #49032 (#49086)
Cherry-picked from #49032

Co-authored-by: zclllyybb <zhaochangle@selectdb.com>
2025-03-16 20:16:40 +08:00
be3e2f36d0 branch-2.1: [fix](nereids)fix the cascadesContext.getMemo()==null #48771 (#49095)
Cherry-picked from #48771

Co-authored-by: zhangm365 <mzhang2048@gmail.com>
2025-03-15 12:20:02 +08:00
afb143c1e8 [improve](binlog) Add config to control whether enable persistent connection during ingesting (#49005)
Cherry-pick #48467, #48761
2025-03-15 10:45:53 +08:00
1eff4538aa branch-2.1: [fix](nereids) fix convert to date literal throw exception #48980 (#48985) 2025-03-15 10:44:40 +08:00
3b70606c4f [improvement](statistics)Improve analyze partition column and key column corner case. (#48757) (#49101)
backport: https://github.com/apache/doris/pull/48757
2025-03-15 10:23:36 +08:00
8589db3ec3 Revert "branch-2.1: [fix](nereids) fix distinct window compute wrong result (#48987) (#49010)" (#49079)
Revert "branch-2.1: [fix](nereids) fix distinct window compute wrong
result (#48987) (#49010)"
2025-03-15 00:02:10 +08:00
d0f6edd212 branch-2.1: [improvement](statistics)Add session variable for partition sample count. #48218 (#49092)
Cherry-picked from #48218

Co-authored-by: James <lijibing@selectdb.com>
2025-03-14 20:43:00 +08:00
ed3a979904 [feature](statistics)Support enable/disable auto analyze for table. (#39532) (#48863)
backport: https://github.com/apache/doris/pull/39532
2025-03-14 17:43:13 +08:00
e5a2b0eea8 Revert "[cherry-pick](jsonb) add a check for jsonb value to avoid invalid jsonb value write into segment file " (#49058)
Reverts apache/doris#48729
temp revert this pr for
PartialUpdateInfo::_generate_default_values_for_missing_cids using empty
string , which will make this check fail.
2025-03-14 17:41:06 +08:00
b5c85e0962 branch-2.1: [fix](hudi) replace non thread safe SimpleDateFormat #48923 (#49022)
Cherry-picked from #48923

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-03-14 17:31:04 +08:00
5988495576 [fix](statistics)Control memory use for sample partition column and key column. (#46534) (#48927)
backport: https://github.com/apache/doris/pull/46534
2025-03-14 17:18:19 +08:00
ad6cf63a28 branch-2.1: [opt](inverted index) uniform profile naming convention #48826 (#48975)
Cherry-picked from #48826

Co-authored-by: zzzxl <yangsiyu@selectdb.com>
2025-03-14 14:04:46 +08:00
79595ad62f [fix](minor) Reorder incorrect logging (#49001) 2025-03-14 12:03:34 +08:00
c989ac9467 branch-2.1: [chore](binlog) add ingesting/downloading binlog latency metrics #48599 (#49002)
cherry pick from #48599
2025-03-14 11:22:23 +08:00
2febf0de1b branch-2.1: [fix](udf) alias udf skip check enable_java_udf #48843 (#48914) 2025-03-14 11:21:01 +08:00
7d521ce288 branch-2.1: [fix](binlog) avoid adding acqurie_md5 param when enable_download_md5… #48573 (#49004)
cherry pick from #48573
2025-03-14 11:18:51 +08:00
7123bbfecb branch-2.1: [fix](nereids) fix distinct window compute wrong result (#48987) (#49010)
cherry pick from #48987
2025-03-14 10:38:16 +08:00
ed2e1ac34a branch-2.1: [fix](variant) update least common type in ColumnObject::pop_back #48935 (#48979)
Cherry-picked from #48935

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
2025-03-13 17:41:17 +08:00
6d2abf9df3 branch-2.1: [Improvement]Add query start datetime #48906 (#48992)
Cherry-picked from #48906

Co-authored-by: wangbo <wangbo@selectdb.com>
2025-03-13 17:40:25 +08:00
ea59465ec7 branch-2.1: [fix](sql cache) fix prepare statement with sql cache throw NullPointerException (#48902) (#48977)
cherry pick from #48902
2025-03-13 14:35:58 +08:00
3b4be62a86 branch-2.1: Revert "[debug](do not merge) add routine load debug info (#48592)" (#48967) 2025-03-13 11:44:07 +08:00
fe37867846 Revert "branch-2.1:[fix](hudi) Fix Memory Leak in BitCaskDiskMap Due to Circular Reference" (#48974)
Reverts apache/doris#48916
This is under verification, revert it now
2025-03-12 19:07:49 +08:00
e40f432cf1 branch-2.1: [fix](case) fix udf cases #48969 (#48972)
Cherry-picked from #48969

Co-authored-by: Dongyang Li <lidongyang@selectdb.com>
2025-03-12 17:35:25 +08:00
fd97f4096b branch-2.1: [fix](auth)fix check internal priv when drop db of external catalog (#48644)
### What problem does this PR solve?
fix check internal priv when drop db of external catalog

It was not picked from the master because the master cannot use the old
optimizer, so it is not fixed in the master
2025-03-12 16:26:49 +08:00
216e9d4fcc branch-2.1:[fix](hudi) Fix Memory Leak in BitCaskDiskMap Due to Circular Reference (#48916)
### What problem does this PR solve?

Problem Summary:
As this issue describes https://github.com/apache/hudi/issues/12918
While debugging a memory leak issue, I noticed a steady increase in the
number of BitCaskDiskMap instances through jmap logs. Using VisualVM, I
found that a memory leak may exist in BitCaskDiskMap due to a circular
reference.

![image](https://github.com/user-attachments/assets/0af6fa66-4c61-46dc-9c67-df1ae18c6607)
- The cleanup method is an instance method that implicitly depends on
the this pointer
- When cleanup is called from shutdownThread, it holds a reference to
this
- Meanwhile, the DiskMap class holds a reference to shutdownThread

![image](https://github.com/user-attachments/assets/3a762e6d-4ebe-4177-a6ff-c01afa5b8180)

This creates a circular reference:
```text
  DiskMap (this) -> shutdownThread -> cleanup -> DiskMap (this)
```
2025-03-12 16:14:56 +08:00
428f6ed425 [test](mtmv) Modify un supported external table test case in branch 2.1 (#48937) 2025-03-12 16:13:54 +08:00
ef4101b52a branch-2.1: [fix](cold hot separation) Fix the issue that files on the remote storage are not deleted after triggering cold data compaction. (#48109) (#48446)
Auto pick #48109 branch 2.1
2025-03-12 15:37:00 +08:00
81f98e007d 2.1.9-rc01 (#48940) 2025-03-11 22:21:08 +08:00
e455bceb91 [fix](function) fix error result when STR_TO_DATE input all space (#4… (#48920)
…8872)
https://github.com/apache/doris/pull/48872
before
```
mysql> select STR_TO_DATE ('  ', '%Y-%m-%d %H:%i:%s');
+-----------------------------------------+
| STR_TO_DATE ('  ', '%Y-%m-%d %H:%i:%s') |
+-----------------------------------------+
|                                         |
+-----------------------------------------+
```
now
```
mysql> select STR_TO_DATE ('  ', '%Y-%m-%d %H:%i:%s');
+-----------------------------------------+
| STR_TO_DATE ('  ', '%Y-%m-%d %H:%i:%s') |
+-----------------------------------------+
| NULL                                    |
+-----------------------------------------+
```

Problem Summary:

None

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-03-11 19:30:38 +08:00
bc3e93a8a9 Revert "branch-2.1: [bug](auto partition) Fix be crash with single replica insert" (#48926)
Reverts apache/doris#48536

BE core after this  PR, revert it。
*** Check failure stack trace: ***
    @     0x564fa82e5606  google::LogMessage::SendToLog()
    @     0x564fa82e2050  google::LogMessage::Flush()
    @     0x564fa82e5e49  google::LogMessageFatal::~LogMessageFatal()
    @     0x564fa9dcb44a  (unknown)
@ 0x564fa8a19e05 google::protobuf::internal::LogMessage::Finish()
    @     0x564f9e68d49e  google::protobuf::Map<>::at<>()
    @     0x564f9e68b805  doris::TabletsChannel::_commit_txn()
    @     0x564f9e68b20b  doris::TabletsChannel::close()
    @     0x564f9e591fee  doris::LoadChannel::_handle_eos()
    @     0x564f9e591ca2  doris::LoadChannel::add_batch()
    @     0x564f9e58c300  doris::LoadChannelMgr::add_batch()
    @     0x564f9e702bb1  std::_Function_handler<>::_M_invoke()
    @     0x564f9e71d3db  doris::WorkThreadPool<>::work_thread()
    @     0x564fab0ad760  execute_native_thread_routine
    @     0x7efd907a8ac3  (unknown)
    @     0x7efd9083a850  (unknown)
    @              (nil)  (unknown)
2025-03-11 17:44:34 +08:00
2c358a1464 branch-2.1: [fix](Nereids) fix split part with regex not exist in source string #48895 (#48910)
Cherry-picked from #48895

Co-authored-by: LiBinfeng <libinfeng@selectdb.com>
2025-03-11 17:37:50 +08:00
90f7f642b6 branch-2.1: [improve](load) improve error message in commit #48316 (#48891)
Cherry-picked from #48316

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
2025-03-11 16:50:28 +08:00
bffa08454e branch-2.1: [opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782 (#48856)
Cherry-picked from #48782

Co-authored-by: seawinde <wusi@selectdb.com>
2025-03-11 15:52:11 +08:00
59b63fe388 branch-2.1: [fix](Nereids) string constant folding process regex delim by mistake #48783 (#48876)
pick from master #48783
2025-03-11 12:13:12 +08:00
b3230c0262 branch-2.1: [fix](ob catalog) fix oceanbase catalog get connection #48593 (#48809)
Cherry-picked from #48593

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-03-11 12:11:59 +08:00
78c9f737d3 branch-2.1: [fix](nereids) fix cast literal to date like throw exception (#48780)
### What problem does this PR solve?

when nereids cast invalid date literal to date like type, it will throws
exceptions:

```
select '' = cast('2020-10-20' as date);
(1105, 'errCode = 2, detailMessage = date/datetime literal [] is invalid')
```

But old planner will not throw exceptions, so let neredis don't throw
exceptions too.

This PR is pick code from: #40202

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-03-11 12:10:38 +08:00
1462dc50d4 branch-2.1: [bug](auto partition) Fix be crash with single replica insert (#48536)
### What problem does this PR solve?

pick:#48101
2025-03-11 12:09:58 +08:00
4dbf3eb12b branch-2.1-pick: [Fix](merge-on-write) should re-calculate delete bitmaps between segments if BE restart before publish (#48775) (#48873)
pick https://github.com/apache/doris/pull/48775
2025-03-11 12:05:45 +08:00
7ca00c114b [fix](load) add lock for runtime_state->tablet_commit_infos (#48709) (#48850)
backport #48709
2025-03-11 12:03:39 +08:00
2c52a7044a [chore](GA)Create CODEOWNERS (#48871) 2025-03-10 18:04:35 +08:00
df1ce09d51 Revert "[fix](Prepared Statment) Fix exec prepared insert stmt in non master error" (#48868)
Reverts apache/doris#48740
Not add test case, should temp revert it and should pick case from
master branch and merge it to 2.1
2025-03-10 17:43:10 +08:00
738dbf0347 branch-2.1: [enhancement](maxcompute)support maxcompute timestamp column type. #48768 (#48842)
Cherry-picked from #48768

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-03-10 17:38:20 +08:00
fe35304aa3 branch-2.1 [fix](nereids) avoid generate Runtime filter whose target is not a base table column (#48779)
### What problem does this PR solve?
pick #48804
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-03-10 17:32:52 +08:00
Pxl
87cb5b8d2c [Chore](case) make test_inject_send_filter_size_fail stable (#48848)
http://43.132.222.7:8111/test/2192182875449727345?currentProjectId=Doris_DorisRegression&expandTestHistoryChartSection=true
2025-03-10 15:57:15 +08:00