Commit Graph

10083 Commits

Author SHA1 Message Date
a461ff3c8a branch-2.1: [fix](load) fix concurrent read and write to runningTxnIds #51615 (#51640)
Cherry-picked from #51615

Co-authored-by: hui lai <laihui@selectdb.com>
2025-06-17 09:26:33 +08:00
ee09f55ca2 branch-2.1: [fix](mtmv)Fix issue that refreshState turns to init when replaying alterStatus editLog #51439 (#51544)
Cherry-picked from #51439

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-06-16 17:30:14 +08:00
03b2606696 [opt](catalog) throw explict error when operation is not supported (#51589) (#51723)
bp #51589
2025-06-16 17:09:14 +08:00
3a1e95c6c2 branch-2.1: [improvement](jdbc catalog) Optimize the acquisition of indentity type in SQLServer (#51659)
pick #51285
2025-06-16 16:50:37 +08:00
eaa6556721 branch-2.1: [fix](nereids) fix cast ipv4 to string #51546 (#51738)
cherry pick from #51546
2025-06-16 15:45:39 +08:00
c6ff4994f3 branch-2.1: [fix](hudi) check requiredFields exists in splitHudiColumnNames in HudiJniScanner #51430 (#51504)
Cherry-picked from #51430

Co-authored-by: Socrates <suyiteng@selectdb.com>
2025-06-16 14:54:40 +08:00
c9623c32f0 branch-2.1: [Bug](sort) fix wrong result coz push down sort with null last #51472 (#51507)
Cherry-picked from #51472

Co-authored-by: Pxl <xl@selectdb.com>
2025-06-16 14:51:12 +08:00
1ee58b216b [fix](test) fix fe UT failed because of timeout (#51699)
fix fe UT failed because of timeout
2025-06-16 09:50:51 +08:00
8734c3fb3b branch-2.1: [improvement](statistics)Agg table set preagg on when doing sample analyzing. (#49918) (#51675)
backport: https://github.com/apache/doris/pull/49918
2025-06-16 09:50:16 +08:00
1a1649376f branch-2.1:[improvement](statistics)Eliminate null values while sample analyzing ndv. (#50574) (#51648)
backport: https://github.com/apache/doris/pull/50574
2025-06-13 15:38:11 +08:00
d4a5039375 branch-2.1: [chore](dep)bump up parquet to 1.15.2 #51372 (#51602)
Cherry-picked from #51372

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
2025-06-13 14:04:55 +08:00
831d2c148a branch-2.1:[enhance](mtmv)cache table snapshot in refresh context (#50855) (#51493)
pick: https://github.com/apache/doris/pull/50855
2025-06-05 12:07:10 +08:00
ffb6fd44b5 branch-2.1: [bugfix](fold) Skip constant folding in from_base64 function to fix binary data handling #50453 (#51413)
Cherry-picked from #50453

Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com>
2025-06-03 14:05:27 +08:00
8cb499ec7c branch-2.1: [opt](audit) add new variable audit_plugin_max_insert_stmt_length to limit the length of insert stmt #51314 (#51391)
Cherry-picked from #51314

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-06-02 09:32:36 +08:00
c22db94c3a [opt](mtmv) optimize mtmv rewrite performance (#49514) (#51309) 2025-06-02 09:21:10 +08:00
ee74a4062c branch-2.1: [Fix] (nereids) Fix incorrect hive references in LogicalIcebergTableSink and update PlanType #49306 (#51287)
Cherry-picked from #49306

Co-authored-by: nivane <1185807945@qq.com>
Co-authored-by: zhaolipan <zhaolipan@learnable.ai>
2025-06-02 00:12:15 +08:00
06a6d0c5e9 branch-2.1: [fix](agent) cancel agent task when it is rejected by agent-task-pool #51138 (#51212)
Cherry-picked from #51138

Co-authored-by: walter <maochuan@selectdb.com>
2025-05-28 17:24:15 +08:00
25f5d5fb1e [fix](nereids) fix colocate agg + join compute wrong result (#48934) (#51313)
cherry pick from #48934
2025-05-28 15:30:58 +08:00
4e3700eccf [branch-2.1: [Fix](Catalog)Ensure preExecutionAuthenticator is properly initialized (#51279)
[branch-2.1: [Fix](Catalog)Ensure preExecutionAuthenticator is properly
initialized


cherry pick https://github.com/apache/doris/pull/50839
https://github.com/apache/doris/pull/50623

---------

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
Co-authored-by: lik40 <lik40@chinatelecom.cn>
2025-05-28 14:28:41 +08:00
7d2134dade branch-2.1: [fix](arrow) Rollback Arrow version 19.0.1 -> 17.0.0, Fix MacOS compile error and decimal type error when convert to Parquet (#51291)
pick #51217

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-27 21:26:11 +08:00
c838678b18 [bugfix](nerids) complete the implementation of the concat method. (#51141) #51259 (#51273)
cherry-pick #51141

---------

Co-authored-by: weiwh1 <weiwh1@chinatelecom.cn>
2025-05-27 18:12:00 +08:00
73ee4c9090 branch-2.1: [chore](cataglog) Unlimit db data size quota (#51108) (#51235)
we should not limit db data size for the db especially for the log
scenarios
doc https://github.com/apache/doris-website/pull/2403

pick #51108
2025-05-27 13:13:45 +08:00
701fc5fa2c [fix](export) remove task from map finally (#51171) (#51207)
bp #51171
2025-05-24 17:45:47 +08:00
6513f9e01f [cherry-pick](branch-21)add some check for udf when result is null (#51084) (#51196) 2025-05-23 20:43:31 +08:00
16521017ad [fix](iceberg)Table operations are not supported for catalogs of the dlf type for 2.1 #50696 (#51112)
bp: #50696
2025-05-23 13:33:10 +08:00
fc9beac468 branch-2.1: [fix](arrow-flight-sql) Separate arrow-flight-sql connection and mysql connection (#51110) 2025-05-22 20:39:48 +08:00
3bc24da9c2 branch-2.1: [fix](meta) fix timeout for syncing image when FE scales out #51120 (#51149)
Cherry-picked from #51120

Co-authored-by: yagagagaga <zhangminke@selectdb.com>
2025-05-22 20:21:49 +08:00
9e0e40c1e6 branch-2.1: [fix](nereids) Not use rule FOUR_PHASE_AGGREGATE_WITH_DISTINCT_WITH_FULL_DISTRIBUTE when mustUseMultiDistinctAgg #51099 (#51140)
Cherry-picked from #51099

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2025-05-22 17:45:15 +08:00
5bc1618c2d branch-2.1: [fix](Nereids) could not work well when check precision for null literal #50815 (#50899)
Cherry-picked from #50815

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-05-22 14:38:39 +08:00
81b5efa560 branch-2.1: [fix](planner)comparing partition column name should be case insensitive when pruning partition in DeleteJob #50674 (#51114)
Cherry-picked from #50674

Co-authored-by: starocean999 <lichi@selectdb.com>
2025-05-22 14:30:12 +08:00
0287bc01c4 branch-2.1: [fix](ddl) Wrong result of scalar type toString for DATETIMEV2 #50237 (#50905)
Cherry-picked from #50237

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2025-05-21 14:22:14 +08:00
16b3e5ff18 [improve](information schema) introduce routine load job system table (#48963) (#49286)
pick #48963

Part IV of https://github.com/apache/doris/issues/48511

doc https://github.com/apache/doris-website/pull/2196

**Introduce routine load job statistic system table:**
```
mysql> show create table information_schema.routine_load_job\G
*************************** 1. row ***************************
       Table: routine_load_job
Create Table: CREATE TABLE `routine_load_job` (
  `JOB_ID` text NULL,
  `JOB_NAME` text NULL,
  `CREATE_TIME` text NULL,
  `PAUSE_TIME` text NULL,
  `END_TIME` text NULL,
  `DB_NAME` text NULL,
  `TABLE_NAME` text NULL,
  `STATE` text NULL,
  `CURRENT_TASK_NUM` text NULL,
  `JOB_PROPERTIES` text NULL,
  `DATA_SOURCE_PROPERTIES` text NULL,
  `CUSTOM_PROPERTIES` text NULL,
  `STATISTIC` text NULL,
  `PROGRESS` text NULL,
  `LAG` text NULL,
  `REASON_OF_STATE_CHANGED` text NULL,
  `ERROR_LOG_URLS` text NULL,
  `USER_NAME` text NULL,
  `CURRENT_ABORT_TASK_NUM` int NULL,
  `IS_ABNORMAL_PAUSE` boolean NULL
) ENGINE=SCHEMA;
1 row in set (0.00 sec)
```

**There are some benefits to empower job with SQL query capability for
statistical information:**

- It can be used in conjunction with metrics add through
https://github.com/apache/doris/pull/48209 to roughly locate abnormal
jobs when Grafana alarms, and the following SQL can be used:

```
SELECT JOB_NAME
FROM information_schema.routine_load_job_statistics
WHERE CURRENT_ABORT_TASK_NUM > 0
   OR IS_ABNORMAL_PAUSE = TRUE;
```

- User can use the `select * from information_schema.routine_load_job`
instead of the `show routine load`. The advantage is that the `show
routine load` can only be searched by name, but SQL can be very flexible
in locating jobs

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-21 12:10:34 +08:00
d63815fa57 branch-2.1: [fix](mv) refresh failed while open enable_single_replica_insert #50986 (#51021)
Cherry-picked from #50986

Co-authored-by: camby <cambyzhu@tencent.com>
2025-05-21 12:07:48 +08:00
6fe7c9087e branch-2.1: [hotfix](jdbc catalog) Fix jdbcclient repeated initialization (#51038)
pick #51036
2025-05-21 12:05:30 +08:00
b1a71fab2c branch-2.1: [fix](pipelinex) fix null aware left anti join instance num #51053 (#51067)
cherry pick from #51053
2025-05-21 11:51:20 +08:00
edb8a51414 branch-2.1: [fix](nereids) fix create view use null literal #49881 (#51006)
Cherry-picked from #49881

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2025-05-19 15:11:32 +08:00
52fa3b5784 branch-2.1: [fix](mtmv)before mtmv refresh,check column type if change (#50730)
pick: https://github.com/apache/doris/pull/49041
2025-05-19 10:30:37 +08:00
967adc3ce1 branch-2.1: [feature](restore) introduce AgentBoundedBatchTask to manage concurrent restore tasks #50740 (#50845)
Cherry-picked from #50740

---------

Co-authored-by: walter <maochuan@selectdb.com>
Co-authored-by: wubiao02 <wubiao02@meituan.com>
2025-05-17 17:23:04 +08:00
fd378a303b branch-2.1: [Fix](external catalog) where tables in the information_schema could not be displayed #49607 (#50878)
Cherry-picked from #49607

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: garenshi <garenshi@tencent.com>
2025-05-17 16:21:11 +08:00
35af34f1ce branch-2.1: [fix](binlog) Record rollup index info for alterJob binlog #50850, #50337 (#50874)
cherry pick from #50850, #50337

---------

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2025-05-17 16:19:57 +08:00
e3fbf39722 branch-2.1: [fix](jdbc catalog) fix a jdbc catalog npe #50901 (#50928)
Cherry-picked from #50901

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-05-17 16:16:00 +08:00
13fbc9efa6 branch-2.1: [fix](hive) fix write hive partition by Doris #50864 (#50921)
Cherry-picked from #50864

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2025-05-17 16:14:23 +08:00
040cacd635 branch-2.1: [fix](cooldown) allow cooldown_ttl = 0 when altering storage policy #50830 (#50856)
Cherry-picked from #50830

Co-authored-by: Luwei <luwei@selectdb.com>
2025-05-17 09:17:33 +08:00
82d1375dc5 branch-2.1: [fix](Nereids) we should also push down expr in join's mark conjuncts #50886 (#50955)
Cherry-picked from #50886

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-05-15 22:46:54 +08:00
01f70deb8b branch-2.1: [fix](nereids) fix parse date time exception #50810 (#50900)
cherry pick from #50810
2025-05-14 23:07:15 +08:00
c4d0e1e693 branch-2.1: [fix](job scheduler) specifies both startTime and immediate, it will trigger one fewer task execution #50624 (#50897)
Cherry-picked from #50624

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-05-14 22:59:48 +08:00
c4e2f05563 branch-2.1: [fix](ut) fix unstable FE ut case for schema change job #50694 (#50887)
Cherry-picked from #50694

Co-authored-by: airborne12 <jiangkai@selectdb.com>
2025-05-14 21:16:20 +08:00
7459764650 [regression-test](fuzzy) accelerate tests by enlarge batch_size in fuzzy (#50861) 2025-05-13 21:44:53 +08:00
9db7a46536 branch-2.1: [bugfix](nerids) align locate function behavior with BE side #50797 (#50832)
Cherry-picked from #50797

Co-authored-by: XLPE <crykix@gmail.com>
2025-05-13 15:19:21 +08:00
0efd97055d branch-2.1: [fix](jdbc catalog) Improve conjunct expression handling in JdbcScanNode #50542 (#50648)
Cherry-picked from #50542

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-05-10 08:38:45 +08:00