Commit Graph

10065 Commits

Author SHA1 Message Date
4e3700eccf [branch-2.1: [Fix](Catalog)Ensure preExecutionAuthenticator is properly initialized (#51279)
[branch-2.1: [Fix](Catalog)Ensure preExecutionAuthenticator is properly
initialized


cherry pick https://github.com/apache/doris/pull/50839
https://github.com/apache/doris/pull/50623

---------

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
Co-authored-by: lik40 <lik40@chinatelecom.cn>
2025-05-28 14:28:41 +08:00
7d2134dade branch-2.1: [fix](arrow) Rollback Arrow version 19.0.1 -> 17.0.0, Fix MacOS compile error and decimal type error when convert to Parquet (#51291)
pick #51217

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-27 21:26:11 +08:00
c838678b18 [bugfix](nerids) complete the implementation of the concat method. (#51141) #51259 (#51273)
cherry-pick #51141

---------

Co-authored-by: weiwh1 <weiwh1@chinatelecom.cn>
2025-05-27 18:12:00 +08:00
73ee4c9090 branch-2.1: [chore](cataglog) Unlimit db data size quota (#51108) (#51235)
we should not limit db data size for the db especially for the log
scenarios
doc https://github.com/apache/doris-website/pull/2403

pick #51108
2025-05-27 13:13:45 +08:00
701fc5fa2c [fix](export) remove task from map finally (#51171) (#51207)
bp #51171
2025-05-24 17:45:47 +08:00
6513f9e01f [cherry-pick](branch-21)add some check for udf when result is null (#51084) (#51196) 2025-05-23 20:43:31 +08:00
16521017ad [fix](iceberg)Table operations are not supported for catalogs of the dlf type for 2.1 #50696 (#51112)
bp: #50696
2025-05-23 13:33:10 +08:00
fc9beac468 branch-2.1: [fix](arrow-flight-sql) Separate arrow-flight-sql connection and mysql connection (#51110) 2025-05-22 20:39:48 +08:00
3bc24da9c2 branch-2.1: [fix](meta) fix timeout for syncing image when FE scales out #51120 (#51149)
Cherry-picked from #51120

Co-authored-by: yagagagaga <zhangminke@selectdb.com>
2025-05-22 20:21:49 +08:00
9e0e40c1e6 branch-2.1: [fix](nereids) Not use rule FOUR_PHASE_AGGREGATE_WITH_DISTINCT_WITH_FULL_DISTRIBUTE when mustUseMultiDistinctAgg #51099 (#51140)
Cherry-picked from #51099

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2025-05-22 17:45:15 +08:00
5bc1618c2d branch-2.1: [fix](Nereids) could not work well when check precision for null literal #50815 (#50899)
Cherry-picked from #50815

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-05-22 14:38:39 +08:00
81b5efa560 branch-2.1: [fix](planner)comparing partition column name should be case insensitive when pruning partition in DeleteJob #50674 (#51114)
Cherry-picked from #50674

Co-authored-by: starocean999 <lichi@selectdb.com>
2025-05-22 14:30:12 +08:00
0287bc01c4 branch-2.1: [fix](ddl) Wrong result of scalar type toString for DATETIMEV2 #50237 (#50905)
Cherry-picked from #50237

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2025-05-21 14:22:14 +08:00
16b3e5ff18 [improve](information schema) introduce routine load job system table (#48963) (#49286)
pick #48963

Part IV of https://github.com/apache/doris/issues/48511

doc https://github.com/apache/doris-website/pull/2196

**Introduce routine load job statistic system table:**
```
mysql> show create table information_schema.routine_load_job\G
*************************** 1. row ***************************
       Table: routine_load_job
Create Table: CREATE TABLE `routine_load_job` (
  `JOB_ID` text NULL,
  `JOB_NAME` text NULL,
  `CREATE_TIME` text NULL,
  `PAUSE_TIME` text NULL,
  `END_TIME` text NULL,
  `DB_NAME` text NULL,
  `TABLE_NAME` text NULL,
  `STATE` text NULL,
  `CURRENT_TASK_NUM` text NULL,
  `JOB_PROPERTIES` text NULL,
  `DATA_SOURCE_PROPERTIES` text NULL,
  `CUSTOM_PROPERTIES` text NULL,
  `STATISTIC` text NULL,
  `PROGRESS` text NULL,
  `LAG` text NULL,
  `REASON_OF_STATE_CHANGED` text NULL,
  `ERROR_LOG_URLS` text NULL,
  `USER_NAME` text NULL,
  `CURRENT_ABORT_TASK_NUM` int NULL,
  `IS_ABNORMAL_PAUSE` boolean NULL
) ENGINE=SCHEMA;
1 row in set (0.00 sec)
```

**There are some benefits to empower job with SQL query capability for
statistical information:**

- It can be used in conjunction with metrics add through
https://github.com/apache/doris/pull/48209 to roughly locate abnormal
jobs when Grafana alarms, and the following SQL can be used:

```
SELECT JOB_NAME
FROM information_schema.routine_load_job_statistics
WHERE CURRENT_ABORT_TASK_NUM > 0
   OR IS_ABNORMAL_PAUSE = TRUE;
```

- User can use the `select * from information_schema.routine_load_job`
instead of the `show routine load`. The advantage is that the `show
routine load` can only be searched by name, but SQL can be very flexible
in locating jobs

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-21 12:10:34 +08:00
d63815fa57 branch-2.1: [fix](mv) refresh failed while open enable_single_replica_insert #50986 (#51021)
Cherry-picked from #50986

Co-authored-by: camby <cambyzhu@tencent.com>
2025-05-21 12:07:48 +08:00
6fe7c9087e branch-2.1: [hotfix](jdbc catalog) Fix jdbcclient repeated initialization (#51038)
pick #51036
2025-05-21 12:05:30 +08:00
b1a71fab2c branch-2.1: [fix](pipelinex) fix null aware left anti join instance num #51053 (#51067)
cherry pick from #51053
2025-05-21 11:51:20 +08:00
edb8a51414 branch-2.1: [fix](nereids) fix create view use null literal #49881 (#51006)
Cherry-picked from #49881

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2025-05-19 15:11:32 +08:00
52fa3b5784 branch-2.1: [fix](mtmv)before mtmv refresh,check column type if change (#50730)
pick: https://github.com/apache/doris/pull/49041
2025-05-19 10:30:37 +08:00
967adc3ce1 branch-2.1: [feature](restore) introduce AgentBoundedBatchTask to manage concurrent restore tasks #50740 (#50845)
Cherry-picked from #50740

---------

Co-authored-by: walter <maochuan@selectdb.com>
Co-authored-by: wubiao02 <wubiao02@meituan.com>
2025-05-17 17:23:04 +08:00
fd378a303b branch-2.1: [Fix](external catalog) where tables in the information_schema could not be displayed #49607 (#50878)
Cherry-picked from #49607

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: garenshi <garenshi@tencent.com>
2025-05-17 16:21:11 +08:00
35af34f1ce branch-2.1: [fix](binlog) Record rollup index info for alterJob binlog #50850, #50337 (#50874)
cherry pick from #50850, #50337

---------

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2025-05-17 16:19:57 +08:00
e3fbf39722 branch-2.1: [fix](jdbc catalog) fix a jdbc catalog npe #50901 (#50928)
Cherry-picked from #50901

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-05-17 16:16:00 +08:00
13fbc9efa6 branch-2.1: [fix](hive) fix write hive partition by Doris #50864 (#50921)
Cherry-picked from #50864

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2025-05-17 16:14:23 +08:00
040cacd635 branch-2.1: [fix](cooldown) allow cooldown_ttl = 0 when altering storage policy #50830 (#50856)
Cherry-picked from #50830

Co-authored-by: Luwei <luwei@selectdb.com>
2025-05-17 09:17:33 +08:00
82d1375dc5 branch-2.1: [fix](Nereids) we should also push down expr in join's mark conjuncts #50886 (#50955)
Cherry-picked from #50886

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-05-15 22:46:54 +08:00
01f70deb8b branch-2.1: [fix](nereids) fix parse date time exception #50810 (#50900)
cherry pick from #50810
2025-05-14 23:07:15 +08:00
c4d0e1e693 branch-2.1: [fix](job scheduler) specifies both startTime and immediate, it will trigger one fewer task execution #50624 (#50897)
Cherry-picked from #50624

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-05-14 22:59:48 +08:00
c4e2f05563 branch-2.1: [fix](ut) fix unstable FE ut case for schema change job #50694 (#50887)
Cherry-picked from #50694

Co-authored-by: airborne12 <jiangkai@selectdb.com>
2025-05-14 21:16:20 +08:00
7459764650 [regression-test](fuzzy) accelerate tests by enlarge batch_size in fuzzy (#50861) 2025-05-13 21:44:53 +08:00
9db7a46536 branch-2.1: [bugfix](nerids) align locate function behavior with BE side #50797 (#50832)
Cherry-picked from #50797

Co-authored-by: XLPE <crykix@gmail.com>
2025-05-13 15:19:21 +08:00
0efd97055d branch-2.1: [fix](jdbc catalog) Improve conjunct expression handling in JdbcScanNode #50542 (#50648)
Cherry-picked from #50542

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-05-10 08:38:45 +08:00
79056d4d7a branch-2.1: [feat](hive) add catalog level partition cache property #50724 (#50762)
Cherry-picked from #50724

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-05-09 22:01:49 +08:00
fee5d40e07 [fix](planner) fix show variable display wrong enable_nereids_planner value (#50746)
### What problem does this PR solve?

fix show variable display wrong enable_nereids_planner value, introduced
by #49913

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-09 17:09:07 +08:00
04cbb2ac66 [fix](information_schema) fix backend_active_tasks table only return one backend's data (#50721) (#50722)
cherry pick from #50721
2025-05-09 15:01:22 +08:00
fde8d05f5d branch-2.1 [opt](nereids) catch all exceptions in StatsCalculator (#49415) (#50364) 2025-05-09 11:24:18 +08:00
0347e8a3c6 branch-2.1: proper planning of shadow columns for load and schema change concurrent execution (#49332) (#50710) 2025-05-09 11:15:08 +08:00
37f3c8f0c7 [fix](nereids) fix fold constant return wrong scale of datetime type (#50142) (#50716)
cherry pick from #50142
2025-05-09 11:12:07 +08:00
bf5885a8f8 branch-2.1: [opt](Nereids) avoid generate nested alias expr when plan insert values (#50388) 2025-05-09 11:06:24 +08:00
12e6a02158 branch-2.1: [enhance](mtmv)Exclude trigger table support db and catalog #49961 (#50554)
Cherry-picked from #49961

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-05-09 09:22:31 +08:00
523681d58e branch-2.1: [enhance](mtmv)Only restrict MTMV to not allow concurrent insert overwrite execution #48673 (#49965)
Cherry-picked from #48673

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-05-08 19:19:30 +08:00
ebe302cb7e branch-2.1: [fix](nereids) do eliminate constant group by key in normalizeagg #49589 (#50212)
Cherry-picked from https://github.com/apache/doris/pull/49589
2025-05-08 18:52:40 +08:00
995f1e5dc0 branch-2.1:[fix](Nereids) fix regression framework compare issue and fix code point count (#49575) (#50667)
backport: https://github.com/apache/doris/pull/49575

Co-authored-by: LiBinfeng <libinfeng@selectdb.com>
2025-05-08 16:53:02 +08:00
6f84c4d7e2 branch-2.1: [fix](planner) return explicit error msg when falling back to old planner (#50676)
### What problem does this PR solve?

Problem Summary:

When fall back to old planner, the partition pruning of external table
will fail,
and NPE will be throw like:
```
2025-04-14 16:22:33,427 WARN (mysql-nio-pool-19697|237161) [HiveScanNode.getSplits():189] get file split failed for table: dwd_log_fact_channel_track_result_hi
java.lang.NullPointerException: null
        at org.apache.doris.datasource.hive.source.HiveScanNode.getPartitions(HiveScanNode.java:137) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.hive.source.HiveScanNode.getSplits(HiveScanNode.java:171) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.createScanRangeLocations(FileQueryScanNode.java:366) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.doFinalize(FileQueryScanNode.java:222) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.finalize(FileQueryScanNode.java:208) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanNode.finalize(PlanNode.java:722) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.JoinNodeBase.finalize(JoinNodeBase.java:456) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanNode.finalize(PlanNode.java:722) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.JoinNodeBase.finalize(JoinNodeBase.java:456) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanNode.finalize(PlanNode.java:722) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.AggregationNode.finalize(AggregationNode.java:402) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanNode.finalize(PlanNode.java:722) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.SetOperationNode.finalize(SetOperationNode.java:158) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanNode.finalize(PlanNode.java:722) ~[doris-fe.jar:1.2-SNAPSHOT]
```

This error msg is very confusing. So this PR return a more explicit
error msg.

### Release note

None
2025-05-08 10:31:37 +08:00
c637667394 branch-2.1:[fix](Nereids) when considering timezone to unix_timestamp, the boundary condition should changed (#49686) (#50669)
backport: https://github.com/apache/doris/pull/49686

Co-authored-by: LiBinfeng <libinfeng@selectdb.com>
2025-05-08 09:46:31 +08:00
9fe87a56bd [cherry-pick](branch-21) support lead/lag function input column as third params (#49381) (#50653)
### What problem does this PR solve?

cherry-pick from master (#49381)

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-07 23:08:56 +08:00
b9cfdcf46a branch-2.1: [chore](dependency)bump tomcat 8.5.86 to 9.0.104 #50622 (#50645)
Cherry-picked from #50622

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
2025-05-07 23:06:46 +08:00
22d846423d [fix](paimon)Set the target size of the split for 2.1 (#50596)
### What problem does this PR solve?

bp: #50083
2025-05-07 23:05:10 +08:00
6c62f28499 branch-2.1: [fix](RecycleBin) fix race condition issue when do execute ShowCatalogRecycleBinStmt (#35759) (#50654)
bp #35759

Co-authored-by: caiconghui <55968745+caiconghui@users.noreply.github.com>
Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2025-05-07 23:02:16 +08:00
e52511ba1b branch-2.1: [chore](test) add ut for sql cache #49389 (#49549)
Cherry-picked from #49389

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
2025-05-07 18:55:41 +08:00