Commit Graph

21979 Commits

Author SHA1 Message Date
cf3fe579ad [fix](regression-test) fix variant_p2 case to show load failure #52652 (#52686) 2025-07-03 08:27:35 +08:00
ea8ef605a6 [fix](main) check error immediately after flight_server init (#52616) (#52624)
backport #52616
2025-07-02 10:21:38 +08:00
06bbfd1eea branch-2.1: [fix](nereids) fix tanh cosntant folding #52100 (#52527)
Cherry-picked from #52100

Co-authored-by: admiring_xm <86338598+wumeibanfa@users.noreply.github.com>
2025-07-02 10:21:10 +08:00
794982a668 branch-2.1: [fix](type) Fix TYPE_DATETIMEV2 upgrade and downgrade compatibility (#52530)
Introduced from https://github.com/apache/doris/pull/49643
2025-07-02 09:24:48 +08:00
577ef510d8 branch-2.1: [fix](mow) fix potential mem leak for DeleteBitmap::AggCache #52596 (#52613)
Cherry-picked from #52596

Co-authored-by: zhannngchen <zhangchen@selectdb.com>
2025-07-02 09:20:30 +08:00
dc8102bbf4 branch-2.1: [Bug](distinct) fix distinct function with over return error result #51875 (#52313)
Cherry-picked from #51875

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
2025-06-30 09:45:50 +08:00
fecc6ac802 branch-2.1: [fix](nereids) exceed expr limit error not fallback to legacy planner (#52431)
### What problem does this PR solve?

Legacy planner check expr limit is not reliable. When planner expr
change, it need explict call expr.analyse to check the limit.

Nereids planner check expr limit at expr's construct function, it's
reliable.

So nereids don't fallback to legacy planner when met exceed expr limit
error, this can avoid legacy planner forget to check the expr limit and
use a lot of memory.


### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-06-29 08:34:50 +08:00
ed94a6d21a branch-2.1: [fix](docker)Add docker-ps 'sudo' permissions #52395 (#52457)
Cherry-picked from #52395

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
Co-authored-by: wuwenchi.wwc <wuwenchi.wwc@oceanbase.com>
2025-06-28 20:06:51 +08:00
8480691871 branch-2.1: [fix](hive) ignore all hidden dir and files #52286 (#52324)
Cherry-picked from #52286

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-06-28 14:23:27 +08:00
e923acef1b branch-2.1: [Fix](JsonReader) Fix the issue where the null bitmap of the JSON reader was not initialized when the JSON path is specified as '$.’ #52211 (#52268)
Cherry-picked from #52211

Co-authored-by: lihangyu <lihangyu@selectdb.com>
2025-06-28 14:21:38 +08:00
4999816ae6 branch-2.1: [fix](map) fix createMap function #52231 (#52401)
Cherry-picked from #52231

Co-authored-by: amory <wangqiannan@selectdb.com>
2025-06-28 14:19:48 +08:00
93e5431bb9 branch-2.1: [test](mv) Fix mv regression test is not chosen stable by CBO #50690 (#52353)
Cherry-picked #50690
2025-06-28 14:19:14 +08:00
1deb221f8c branch-2.1: [chore](ci)Replace local setup-maven with setup-java for more stable and simplified Maven setup #52229 (#52257)
Cherry-picked from #52229

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
2025-06-28 11:13:19 +08:00
cb60589c0a branch-2.1: [fix](trash) Fix shadow variable causing garbage scheduling time errors #51647 (#51996)
Cherry-picked from #51647

Co-authored-by: deardeng <dengxin@selectdb.com>
2025-06-28 11:12:25 +08:00
c671a09a56 branch-2.1: [fix](iceberg)Ensure proper authentication context before accessing Iceberg Catalog #52149 (#52234)
Cherry-picked from #52149

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
2025-06-28 10:59:03 +08:00
95f10aa8a1 branch-2.1:[fix](mtmv) fix when compatible fail, will throw NPE (#49875) (#52273)
pick: https://github.com/apache/doris/pull/49875
2025-06-28 10:54:44 +08:00
af070ac853 [branch-2.1](timezone) Fix incorrect DST handling (#51454) (#52418)
pick https://github.com/apache/doris/pull/51454
2025-06-28 10:48:58 +08:00
e029e420ae branch-2.1: [fix](nereids) fix calc repeat property stack overflow #52159 (#52188)
cherry pick from #52159
2025-06-28 10:22:48 +08:00
5d20a04b32 branch-2.1: [fix](auth)Only treat admin@% and root@% as system users. #50904 (#52283)
Cherry-picked from #50904

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-06-28 10:11:13 +08:00
86ea0f2b81 [fix](be) Fix be aws-sdk-cpp memory leak (#52185) (#52298)
* https://github.com/aws/aws-sdk-cpp/pull/2314
* In the aws-sdk-cpp code, there is a suggestion: "Please call
ShutdownAPI from the same thread from which InitAPI() has been called"
otherwise it will cause memory leak.

```
=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    #4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    #5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    #6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-06-28 10:10:32 +08:00
d7184af32b branch-2.1: [fix](cluster)fix fe host may be contains Scope Identifier #52076 (#52398)
Cherry-picked from #52076

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-06-28 10:09:22 +08:00
74a409d74e [test](mv)forbid create mv test case (#52350)
pick from #51777 
[test](mv)forbid create mv test case
2025-06-28 10:08:19 +08:00
307c047377 branch-2.1: [fix](Prepared Statment) Fix exec prepared insert stmt in non master error (#48689) (#52265)
backport: https://github.com/apache/doris/pull/48689

Co-authored-by: Lijia Liu <liutang123@yeah.net>
2025-06-28 09:57:53 +08:00
f0ce208e68 [fix](test) Create mv but not check finished (#52167) 2025-06-28 09:48:44 +08:00
16777c7235 branch-2.1: [fix](nereids) correct the log class in the command (#43896) (#52253)
backport: https://github.com/apache/doris/pull/43896

Co-authored-by: Yao-MR <yao.mr.cn@gmail.com>
2025-06-25 14:51:30 +08:00
43b99ddd7a [fix](mtmv) Fix compensate union all wrongly when partition mv use partition external table (#52218)
Fix compensate union all wrongly when partition mv use partition
external table

in test mv/external_table/part_partition_invalid.groovy
order_qt_after_modify_data_and_refresh_catalog data should be as
following:
```
1	2	1	2023-10-17	2023-10-17
2	2	2	2023-10-18	2023-10-18
3	2	3	2023-10-19	2023-10-19
```


### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #49514 

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [x] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-06-25 14:50:47 +08:00
6dca968e96 branch-2.1:[fix](planner)Fix the concurrency issue caused by Expr setting member… (#52194)
… variables (#51389)

pick: https://github.com/apache/doris/pull/51389
2025-06-25 11:45:27 +08:00
b3588b01c0 branch-2.1: [improvement](nereids)Support GROUP BY ... WITH ROLLUP syntax #51948 (#51978)
Cherry-picked from #51948

Co-authored-by: James <lijibing@selectdb.com>
2025-06-25 09:59:02 +08:00
042e18068a [chore](test) fix regression-test/suites/fault_injection_p0/test_variant_bloom_filter.groovy (#52201) 2025-06-24 18:23:49 +08:00
93d7eb57bc branch-2.1: [fix](hudi catalog) Fix the Kerberos authentication error when querying hudi table #51713 (#51903)
Cherry-picked from #51713

Co-authored-by: heguanhui <hgh_wy163mail@163.com>
2025-06-24 14:20:47 +08:00
667510f71b [branch-2.1]Remove memory sum check (#52055) 2025-06-24 14:08:28 +08:00
4cb66f3a40 branch-2.1: [chore](http) return NOT_FOUND if dir is not exists #51897 (#52156)
Cherry-picked from #51897

Co-authored-by: walter <maochuan@selectdb.com>
2025-06-24 10:39:28 +08:00
490963634b [enhancement](time_series) increase the version limit for the time series table (#51371) (#52077)
pick master #51371
2025-06-24 09:44:24 +08:00
c9e2df607b [opt](metrics) add metrics pipeline_task_queue_size (#51878) (#52141)
### What problem does this PR solve?

pick #51878 to branch-2.1
2025-06-24 09:27:13 +08:00
347c1e1e63 [branch-2.1]refactor workload group cpu hard limit (#51278) (#51382)
pick #51278
2025-06-23 14:45:45 +08:00
d194014914 [fix](ip) fix ip type in IndexDef support (#50637)
### What problem does this PR solve?
only in branch-2.1  we do not support old indexDef to support Ip type 
and fix array_contains && arrays_overlap for with type array<ip> which
backport: https://github.com/apache/doris/pull/50637
Issue Number: close #xxx
2025-06-20 19:51:38 +08:00
e9744f7845 [Bug](aggregate) fix bitmap_union return error result in query sql (#51866) 2025-06-20 19:50:19 +08:00
a42fe9f6e8 branch-2.1: [fix](metadata) replace view will break view's metadata #51058 (#52043)
cherry-pick from #51058
2025-06-20 19:49:03 +08:00
f58b3204ca branch-2.1: [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials #51245 (#51337)
Cherry-picked from #51245

---------

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
2025-06-20 14:12:25 +08:00
2e8ec1850f [bugfix](memleak) fix memleak in arrow input stream (#51929) (#51992) 2025-06-20 14:07:45 +08:00
614c7f8579 [fix](restore) Analyze idx meta in Olap Table after restore commit (#51883)
reproduce: backup and restore Table or restart fe, then insert


before:
```
desc tbl all
*************************** 26. row ***************************
    IndexName: idx
IndexKeysType: AGG_KEYS
        Field: mv_date_format(hours_add(happen_time, 1), '%Y-%m-%d %H:00:00')
         Type: varchar(65533)
 InternalType: varchar(65533)
         Null: Yes
          Key: true
      Default: NULL
        Extra: 
      Visible: true
   DefineExpr:
  WhereClause:

insert into wrong
[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = column has no source field, column=mva_SUM__CASE WHEN 1 IS NULL THEN 0 ELSE 1 END
```

now
```
desc tbl all
*************************** 26. row ***************************
    IndexName: idx
IndexKeysType: AGG_KEYS
        Field: mv_date_format(hours_add(happen_time, 1), '%Y-%m-%d %H:00:00')
         Type: varchar(65533)
 InternalType: varchar(65533)
         Null: Yes
          Key: true
      Default: NULL
        Extra: 
      Visible: true
   DefineExpr: date_format(hours_add(`happen_time`, 1), '%Y-%m-%d %H:00:00')
  WhereClause: (`k1` > '2025-06-12 00:00:00')

insert into ok
```
2025-06-20 14:04:47 +08:00
0230d60e06 branch-2.1:[fix](cooldown)No need to rdlock inside get_cooldown_tablets, there's enough rdlock inside tablet internal function calls.#39211 (#51611)
cherry-pick: #39211
2025-06-20 14:04:00 +08:00
956fb96dcd branch-2.1: [bugfix](query) Fix for incorrect query results of variant type #51651 (#51707)
### What problem does this PR solve?

Cherry-picked from #51651

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-06-20 14:03:04 +08:00
7ef4824575 Pick [enhancement](fe-meta) Support skip specified journal to avoid FE can't be start due to a damaged journal (#35783) (#51877)
If fe meta has damaged journals, fe can't be started successfully.
Adding a configuration to skip the damaged journals.

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [x] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

Co-authored-by: zxealous <zhouchangyue@baidu.com>
2025-06-20 14:01:43 +08:00
a75760d18f brach-2.1 cherry-pick [Fix](Variant) fix serialize with json key contains . as name (#51864)
cherry-pick from #51857
2025-06-20 14:00:00 +08:00
43a5116b9e branch-2.1: [fix](jdbc)make sure init the jdbc client before using it (#51541)
This may cause some statistic collection task fail:
```
2025-06-05 15:02:32,277 WARN (Statistics Job Appender|182) [ExternalCatalog.buildDbForInit():902] Failed to check db DORIS_TEST exist in remote system, ignore it.
java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.jdbc.client.JdbcClient.getDatabaseNameList()" because "this.jdbcClient" is null
        at org.apache.doris.datasource.jdbc.JdbcExternalCatalog.listDatabaseNames(JdbcExternalCatalog.java:267) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.ExternalCatalog.getFilteredDatabaseNames(ExternalCatalog.java:476) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.ExternalCatalog.lambda$makeSureInitialized$0(ExternalCatalog.java:318) ~[doris-fe.jar:1.2-SNAPSHOT]
        at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145) ~[hive-catalog-shade-2.1.4.jar:2.1.4]
        at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139) ~[hive-catalog-shade-2.1.4.jar:2.1.4]
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) ~[hive-catalog-shade-2.1.4.jar:2.1.4]
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916) ~[?:?]
```

Only for branch-2.1.
Fix master in #51471
2025-06-20 13:58:41 +08:00
6cca489262 branch-2.1: [opt](insert) return root cause of exception when executing insert #51721 (#51759)
Cherry-picked from #51721

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-06-20 13:57:34 +08:00
c7a94ba608 branch-2.1: [chore](load) reduce log for data load #51172 (#51801)
cherry pick from #51172
2025-06-20 13:56:00 +08:00
746d83647e [fix](mtmv) Fix compensate union wrongly when direct query is empty relation (#51700) (#51899)
commitId: 85b7231c
pr: https://github.com/apache/doris/pull/51700
2025-06-20 13:53:06 +08:00
8c2713b2e5 [fix](alter) alter partition without storage_policy property will also cancel storage_policy (#51662) (#51910)
### What problem does this PR solve?

pick #51662 to branch-2.1

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-06-20 12:18:36 +08:00