Commit Graph

10001 Commits

Author SHA1 Message Date
b0c8cb0818 branch-2.1: [Bug](function) fix Could not find function explode_json_array_json_outer #50164 (#50227)
Cherry-picked from #50164

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
2025-04-23 15:59:08 +08:00
98f5d95fba branch-2.1: [feat](function) SUBSTRING_INDEX function delimiter supports dynamic #50149 (#50303)
Cherry-picked from #50149

Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com>
2025-04-23 15:00:40 +08:00
a193f16a5a branch-2.1:[fix](auditlog)Set sqlHash in executeInternalQuery (#49984) (#50271)
backport: https://github.com/apache/doris/pull/49984
2025-04-22 18:05:56 +08:00
1efe62c7ba branch-2.1: [opt](hive) add option to get schema from table object #50038 (#50269)
Cherry-picked from #50038

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-04-22 14:25:03 +08:00
b9bb9e1611 branch-2.1: [fix](Export) fix the lock leak issue of Export #50088 (#50219)
Cherry-picked from #50088

Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
2025-04-22 14:21:42 +08:00
1c90fc6c02 branch-2.1: [fix](connect) fix wrong format causing connection fail #50214 (#50217)
bp #50214
2025-04-21 16:34:00 +08:00
188cbabb58 [enhance](auth)Remove restrictions on user creation and other operations when enabling ranger/LDAP (#50137)
### What problem does this PR solve?
- In version 2.1, the global permission check still calls the internal
permission interface. If grant is not allowed, it will be impossible to
assign admin and other permissions to users
- According to the current design of LDAP, if there is no user in LDAP,
Doris will check again to see if the user exists internally. If there
is, login will also be allowed. Therefore, creating users should not be
prohibited
2025-04-21 14:15:07 +08:00
d575a0921a [fix](hive) Fix hive metastore client socket timeout (#50192) 2025-04-21 14:09:20 +08:00
01b5cc4f1f [opt](mtmv) Doesn't throw npe when not set distribution info which should be random (#49402)
### What problem does this PR solve?

pr: https://github.com/apache/doris/pull/39427
commitId: 9ffb060a



Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-04-21 10:41:07 +08:00
481fca6766 [revert](statistics)Still use sample analyze sql when sample full table. (#50199)
Still use sample analyze sql when sample full table.
2025-04-21 10:12:01 +08:00
048537cd03 branch-2.1:[fix](nereids)Use utf-8 when convert string like literal to double. (#50085) (#50155)
backport: https://github.com/apache/doris/pull/50085
2025-04-19 17:16:20 +08:00
967d0a59b9 branch-2.1: [fix](load) reset routine load task EOF sign correctly #50048 (#50179)
Cherry-picked from #50048

Co-authored-by: hui lai <laihui@selectdb.com>
2025-04-19 17:15:18 +08:00
64815c6403 branch-2.1: [fix](dialect)fix starrocks dialect #50099 (#50165)
Cherry-picked from #50099

Co-authored-by: Jiwen liu <61498169+liujiwen-up@users.noreply.github.com>
2025-04-18 20:25:38 +08:00
4146cbbbbb branch-2.1-pick: [Opt](mow) Forbid time_series compaction policy on unique table (#49905) (#50132)
pick https://github.com/apache/doris/pull/49905
2025-04-18 11:34:18 +08:00
77ea907b54 branch-2.1: [Bug](materialized-view) Fixed the problem of using drop table force and create mv stmt at the… #41580 (#50133)
Cherry-picked from #41580

Co-authored-by: Pxl <pxl290@qq.com>
2025-04-18 11:30:47 +08:00
ea29bc523e branch-2.1: [Enhancement](GEO) Support Multipolygon and some spatial functions (#50073)
pick: https://github.com/apache/doris/pull/37003,
https://github.com/apache/doris/pull/48695 and
https://github.com/apache/doris/pull/49665

---------

Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
Co-authored-by: koi <koi20000@163.com>
2025-04-17 09:25:29 +08:00
2ef136cad6 [fix](variant) building index on the variant column is prohibited (#49159) (#50028)
pick from master #49159
2025-04-16 17:52:56 +08:00
bbdb057e2c branch-2.1: [fix](nereids) EliminateGroupByConstant should replace agg's group by after removing constant group by keys #49473 (#50044)
Cherry-picked from #49473

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2025-04-16 13:02:39 +08:00
7ba2f87087 branch-2.1: [fix](binlog) Replaced partition should be record to Dropped Resources #48686 (#49789)
pick: https://github.com/apache/doris/pull/48686/files
2025-04-16 12:41:12 +08:00
b9d447741e Revert "branch-2.1: [Fix](Serde) Support hive compatible output format #49036" (#49986)
Reverts apache/doris#49831
2025-04-11 16:08:02 +08:00
9960b0960e branch-2.1: [bug](auto partition) Fix be crash with single replica insert (#48929)
Something may be wrong in #48536, and it reverts at #48926.

We need to re-run the pipeline and see what was wrong.
2025-04-11 15:23:51 +08:00
ba1ad12a21 branch-2.1: [Fix](Serde) Support hive compatible output format #49036 (#49831)
Cherry-picked from #49036

Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
2025-04-11 15:13:02 +08:00
40a62c3b82 branch-2.1: [feat](dialect) add starrocks dialect #49951 (#49974)
Cherry-picked from #49951

Co-authored-by: Jiwen liu <61498169+liujiwen-up@users.noreply.github.com>
2025-04-11 09:23:06 +08:00
530bad3d14 Set isQuery audit log correctly when fallback to old planner. (#49959) 2025-04-10 23:27:54 +08:00
78b8f31895 branch-2.1 [Fix](Catalog) Close system resources when dropping catalog ( #49621) (#49935)
https://github.com/apache/doris/pull/49621
(cherry picked from commit
902b95b1bf)
2025-04-10 23:23:51 +08:00
e4d0c672a3 branch-2.1: [fix](Nereids) cse extract wrong expression from lambda expressions (#49166) (#49942)
cherry pick from #49166

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-04-10 23:22:55 +08:00
c83c2e18bc [fix](planner) fix fallback to legacy planner compute wrong result (#49913)
### What problem does this PR solve?

doris 3.x not need this fix because the legacy planner already be
deleted

what's wrong:
1. when nereids planner can not parse the sql, doris will fallback to
legacy planner and maybe handle this case, but we not set the
`enable_nereids_planner` to `false` for legacy planner
2. the `ignore_storage_data_distribution` only support when
`enable_nereids_planner` is `true`
3. the legacy planner will use one phase aggregation when input fragment
only has one instance
4. the legacy planner maybe compute one instance by invoke
`OlapScanNode#getNumInstances()` if `enable_nereids_planner == true &&
ignore_storage_data_distribution == true`, even the data place at
multiple backends, then the aggregation will compute wrong result

so when fallback to legacy planner, the legacy planner should set
`enable_nereids_planner` to `false`, then the input OlapScanNode will
not compute one instance, and the AggregationNode will use two phases to
compute the right result.


### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-04-10 23:22:18 +08:00
bfe7469560 branch-2.1: [fix](oss) the write to hive table on oss-hdfs may fail (#49754) (#49911)
bp #49754
2025-04-10 22:01:42 +08:00
4ae9451b63 branch-2.1: [opt](connection) add connection num in error msg (#49471) (#49910)
bp #49471
2025-04-10 22:00:51 +08:00
81ea088c96 branch-2.1: [fix](backup) Save snapshot meta during replay #49550 (#49606)
Cherry-picked from #49550

Co-authored-by: walter <maochuan@selectdb.com>
2025-04-09 14:19:19 +08:00
55974f7fb9 [fix](mtmv) Fix materialized rewrite oom when the num of relation mapping is too large (#48887) (#49850) 2025-04-08 15:18:26 +08:00
11fecd123d branch-2.1: [fix](gson) Missing the serialization of the partition's storage policy #49721 (#49841)
Cherry-picked from #49721

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2025-04-08 11:02:13 +08:00
6974a8fc4a branch-2.1: [opt](nereids)do not apply CSE(Common Sub Expression) upon multiDataSink #33746 (#49797) 2025-04-04 20:49:07 +08:00
342b55afc2 branch-2.1: [fix](binlog) get table with db lock if the table not exists #49566 (#49648)
Cherry-picked from #49566

Co-authored-by: walter <maochuan@selectdb.com>
2025-04-04 20:46:58 +08:00
3a282bd307 branch-2.1:[fix](auth)Delete from should not check select_priv (#49794)
pick: https://github.com/apache/doris/pull/49239
2025-04-04 20:46:43 +08:00
15662c06e6 branch-2.1: [fix](auth)Ignore replay edit log error of auth #49348 (#49787)
Cherry-picked from #49348

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-04-04 20:44:54 +08:00
2b08429577 branch-2.1: [fix](meta) do not check replica allocation when replay #49569 (#49604)
Cherry-picked from #49569

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-04-03 09:50:24 +08:00
3d10db4786 branch-2.1 [opt](nereids) set column stats unkown by default when derive Not expressoin #48864 (#49742) 2025-04-03 09:48:46 +08:00
0735c19cdd branch-2.1: [fix](paimon) Covert Paimon DeletionFile Path to StoragePath in fe #49645 (#49751)
Cherry-picked from #49645

Co-authored-by: Socrates <suyiteng@selectdb.com>
2025-04-02 20:25:10 +08:00
f7f230dd34 branch-2.1: [fix](nereids)canInferNotNullForMarkSlot method get wrong result if fold constant rule is disabled (#49695) 2025-04-02 10:24:34 +08:00
e898dbbba0 branch-2.1: [fix](mc)Fixed the issue that maxcompute catalog can only read part of the timestamp data #49600 (#49706)
Cherry-picked from #49600

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-04-01 17:09:15 +08:00
a9939c09c1 branch-2.1: [improve](thrift) Config thrift_max_message_size for FE SIMPLE and TH… #49678 (#49725)
Cherry-picked from #49678

Co-authored-by: walter <maochuan@selectdb.com>
2025-04-01 17:03:48 +08:00
Pxl
e5fe002805 [Bug](materialized-view) check duplicate expr when create mv stmt not have groupby exprs (#49595) (#49696)
pick from #49595
2025-04-01 15:51:41 +08:00
ec25f7573e branch-2.1: [fix](nereids) project child output to union output in correct order after eliminate empty relation #49257 (#49464)
Cherry-picked from #49257

Co-authored-by: minghong <zhouminghong@selectdb.com>
2025-03-29 20:33:42 +08:00
82064902c0 branch-2.1: [Bug] Fix accidental table deletion during restore job #48820 (#49498)
Cherry-picked from #48820

Co-authored-by: wubiao <biao.wu@aliyun.com>
Co-authored-by: wubiao02 <wubiao02@meituan.com>
2025-03-29 20:26:12 +08:00
e932094a05 branch-2.1: [fix](jdbc catalog) ensure initialization before fetching row count #49442 (#49476)
Cherry-picked from #49442

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-03-29 20:25:09 +08:00
cc0b2585ac branch-2.1: [function](date) Support date trunc function #49540 (#49661) 2025-03-29 20:21:33 +08:00
1259ee5088 branch-2.1: [Feature](function) support year of week #48870 (#49012) 2025-03-29 11:24:45 +08:00
2ab34bfd86 branch-2.1: [fix](catalog)when checkpoint,use cacheThreadPool #49097 (#49518)
Cherry-picked from #49097

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-03-29 10:33:40 +08:00
94986fc574 branch-2.1: [fix](multi-catalog) Fix bug: "Can not create a Path from an empty string" (#49382) (#49641)
### What problem does this PR solve?
Problem Summary:
In HiveMetaStoreCache, the function FileInputFormat.setInputPaths is
used to set input paths. However, this function splits paths using
commas, which is not the expected behavior. As a result, when partition
values contain commas, it leads to incorrect path parsing and potential
errors.
```java
  public static void setInputPaths(JobConf conf, String org.apache.hadoop.shaded.com.aSeparatedPaths) {
    setInputPaths(conf, StringUtils.stringToPath(
                        getPathStrings(org.apache.hadoop.shaded.com.aSeparatedPaths)));
  }
```
To prevent FileInputFormat.setInputPaths from splitting paths by commas,
we use another overloaded version of the method. Instead of passing a
comma-separated string, we explicitly pass a Path object, ensuring that
partition values containing commas are handled correctly.
```java
  public static void setInputPaths(JobConf conf, Path... inputPaths) {
    Path path = new Path(conf.getWorkingDirectory(), inputPaths[0]);
    StringBuffer str = new StringBuffer(StringUtils.escapeString(path.toString()));
    for(int i = 1; i < inputPaths.length;i++) {
      str.append(StringUtils.COMMA_STR);
      path = new Path(conf.getWorkingDirectory(), inputPaths[i]);
      str.append(StringUtils.escapeString(path.toString()));
    }
    conf.set(org.apache.hadoop.shaded.org.apache.hadoop.mapreduce.lib.input.
      FileInputFormat.INPUT_DIR, str.toString());
  }
```

### Release note

None
2025-03-29 09:13:43 +08:00