database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,941 Commits 1 Branch 0 Tags
f2a37d58fb86ef09165ac8d34535926c47517cb3
Commit Graph

5240 Commits

Author SHA1 Message Date
zclllhhjj
f7b6ed772b [Fix-2.1](testcase) Fix some testcases unstable results due to lack of orderby (#39892) 
## Proposed changes

Issue Number: close #xxx

some sql's order-by column has same value, add more order-by column to
fix it.
 2024-08-25 08:45:36 +08:00
LiBinfeng
9997911ec9 [feat](Nereids) support nereids hint position detaction (#39113) (#39417) 
cherry-pick: #39113
When use hint in wrong position or use unsupport hint, use channel(2) to
filter it out

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-24 23:59:54 +08:00
Mingyu Chen
263746b04b [fix](paimon) fix crash when enable cache with paimon deletion vector(#39877) (#39875) 
bp #39877
 2024-08-24 17:58:20 +08:00
zclllhhjj
460605ae3c [branch-2.1] pick some prs (#39860) 
## Proposed changes

Issue Number: close #xxx

https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
 2024-08-24 17:26:42 +08:00
Mingyu Chen
de2e8f0ae6 [fix](ctas) fix NPE when ctas with old planner and varchar issue (#39744) (#39871) 
bp #39744
 2024-08-24 09:24:47 +08:00
Mingyu Chen
b9da934b16 [fix](hive) report error with escape char and null format (#39700) (#39869) 
bp #39700

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-08-24 09:23:03 +08:00
lihangyu
a6f267c479 [pick](Variant) fix element_at should return nullable if result type is nullable (#39846) 
#39732
 2024-08-24 09:22:03 +08:00
Jerry Hu
8cf6c6a2b5 [fix](agg function) incorrect result of map agg(#39743) (#39854) 
## Proposed changes

pick #39743
 2024-08-24 09:19:52 +08:00
morrySnow
9d597bde68 [fix](Nereids) producer to consumer should be multimap in cte (#39850) (#39867) 2024-08-23 23:25:11 +08:00
Mingyu Chen
508c7a7040 [fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. (#39239) (#39865) 
bp #39239

Co-authored-by: daidai <2017501503@qq.com>
 2024-08-23 23:21:02 +08:00
lihangyu
3103bb08dc [pick](Variant) casting to decimal type may lost precision (#39843) 
#39650
 2024-08-23 22:47:32 +08:00
zhangdong
19c51cc217 [fix](mtmv) use name instead of id in meta of MTMV (#39355) (#39748) 
pick from master #39355
 2024-08-23 20:40:54 +08:00
wangbo
6ceb574aa0 [branch-2.1]Pick IO limit/workload group usage table (#39839) 2024-08-23 18:51:47 +08:00
Xinyi Zou
e716658fba [branch-2.1](arrow-flight-sql) Fix exceed user property max connection cause Reach limit of connections (#39836) 
pick #39127
pick #39802
 2024-08-23 17:27:34 +08:00
xzj7019
424ad2384a [opt](nereids) refine left semi/anti cost under short-cut opt (#39636) 
## Proposed changes

pick from https://github.com/apache/doris/pull/37951

---------

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-23 17:26:56 +08:00
Xinyi Zou
baf5b71b39 [branch-2.1](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache (#39829) 
pick #38185
 2024-08-23 17:21:42 +08:00
wuwenchi
c40246efa9 [bugfix](iceberg)Fixed random core with writing iceberg partitioned table for 2.1 (#39808)(#39569) (#39832) 
## Proposed changes

bp: #39808 #39569
 2024-08-23 17:19:48 +08:00
Xujian Duan
8f15efdbb8 [cherry-pick](branch-2.1) fix delete random distributed tbl (#39830) 
## Proposed changes

cherry-pick #37985

<!--Describe your changes.-->
 2024-08-23 17:17:05 +08:00
slothever
67a8099991 [fix](multi-catalog)fix max compute array and map type read offset (#39822) 
bp #39680
 2024-08-23 16:53:52 +08:00
walter
e0b65d0ff4 [fix](regression) Fix creating db for downstream url #39601 (#39818) 
cherry pick from #39601
 2024-08-23 16:15:19 +08:00
Mingyu Chen
1f16daa5f6 Revert "[bugfix](iceberg)clear block for partition values for 2.1 (#39569)" (#39815) 
Reverts apache/doris#39729
 2024-08-23 11:58:42 +08:00
htyoung
6c10c47f79 [fix](fe) LIST partition table support modify default bucket num (#39688) 
## Proposed changes
bp #39696

Issue Number: close #39684

```sql
CREATE TABLE `test1` (
    `id1` VARCHAR(255) NULL COMMENT 'id1',
    `id2` VARCHAR(255) NULL COMMENT 'id2',
    `event_time` VARCHAR(255) NULL COMMENT '事件时间',
    `event_date` VARCHAR(255) NULL COMMENT '事件日期',
    `event_ts` VARCHAR(256) NULL COMMENT '事件发生时间戳(毫秒)',
    `dt` VARCHAR(255) NOT NULL COMMENT '日期分区',
    `hr` VARCHAR(255) NOT NULL COMMENT '小时分区'
  ) ENGINE = OLAP DUPLICATE KEY(`id1`) COMMENT 'xxx' PARTITION BY LIST(`dt`, `hr`) (
    PARTITION p2024082021 VALUES IN (("2024-08-20", "21"))
  ) DISTRIBUTED BY HASH(`dt`, `hr`) BUCKETS 2 PROPERTIES (
    "replication_allocation" = "tag.location.default: 1",
    "min_load_replica_num" = "-1",
    "is_being_synced" = "false",
    "storage_medium" = "hdd",
    "storage_format" = "V2",
    "inverted_index_storage_format" = "V1",
    "light_schema_change" = "true",
    "disable_auto_compaction" = "false",
    "enable_single_replica_compaction" = "false",
    "group_commit_interval_ms" = "10000",
    "group_commit_data_bytes" = "134217728"
  );
```

1. 修改前表已有分区的bucket num为2

![image](https://github.com/user-attachments/assets/77efdd0c-f845-41a4-9a31-e454808ffe67)

2. 修改List分区表bucket num（从2 -> 4）

![image](https://github.com/user-attachments/assets/53b19918-2879-4cb3-b2bd-84ba35a7fc59)

3. 修改List分区后，添加的新分区bucket num为4

![image](https://github.com/user-attachments/assets/1f41f73f-d70f-433e-a7b6-8346b7dfcc4e)

Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>
 2024-08-23 11:52:16 +08:00
zy-kkk
40a58b9e42 [branch-2.1][regression test](jdbc catalog) Enable CLICKHOUSE_ALWAYS_RUN_INITDB_SCRIPTS for clickhouse docker (#39667) 
pick (#39425) #39693
 2024-08-23 09:59:03 +08:00
Mryange
0f8bd33077 [fix](scan) fix predicate contains cast that results in null, the pr… (#39809) 
…edicate will be miss. (#39550)
https://github.com/apache/doris/pull/39550
```
drop table datetest;

create table datetest (
  id int,
  dt date
)
DUPLICATE key (id)
distributed by hash(id) buckets 1
properties(
  "replication_num" = "1"
);
insert into datetest values (1, '2024-01-01');

mysql [test10]>select dt from datetest  WHERE dt = 1 ;
+------------+
| dt         |
+------------+
| 2024-01-01 |
+------------+
```

now

```
mysql [test10]>select dt from datetest  WHERE dt = 1 ;
Empty set (0.16 sec)
```

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 01:46:22 +08:00
wuwenchi
dc732fe33f [bugfix](iceberg)clear block for partition values for 2.1 (#39569) (#39729) 
## Proposed changes

bp: #39569

clear block, or we will get wrong partition values.
 2024-08-22 22:43:02 +08:00
Xinyi Zou
40500f2b10 [branch-2.1](thirdparty) upgrade arrow to 17.0.0 (#39773) 
pick #38572
 2024-08-22 18:35:32 +08:00
TengJianPing
ba7baa7e6b [fix](window_funnel) fix upgrading problem caused by behaviour change of window_funnel (#39766) 
## Proposed changes

Issue Number: close #xxx

For the latest published 2.1 version `2.1.5`:
```
max_be_exec_version=4;
AGG_FUNCTION_NEW=2;
```
and `branch-2.1`:
```
max_be_exec_version=5;
AGG_FUNCTION_NEW=2;
```
It will cause problem when upgrading.
This PR fix the problem, set `AGG_FUNCTION_NEW` to `5`.
 2024-08-22 17:26:51 +08:00
zhangdong
f553645a71 [fix](mtmv) transfer col in mysql varchar to text when create MTMV (#37668) (#39727) 
pick from master #37668
 2024-08-22 15:20:59 +08:00
xzj7019
10f3e88f7a [fix](nereids) fix distribution expr list (#39435) 
pick from #39148
 2024-08-22 15:19:51 +08:00
wangbo
1c566253a8 [Pick][Improment]Query queued by be memory (#37559) (#39733) 
pick #37559
 2024-08-22 15:14:47 +08:00
toms
fd13962015 [chore](nereids) Added compatibility with mysql alias conflict (#38104) (#38440) 
throw table name/alias conflict exception to keep same behavior with mysql

for example:
```sql
select * from test.a b, test.b
```

error:
```
Not unique table/alias: 'b'
```
 2024-08-22 14:37:49 +08:00
toms
50f440e653 [chore](nereids) Added compatibility with mysql alias filter (#39738) 
qt_filter_select4 """
       select * from filter_alias_test.test b where filter_alias_test.b.id = 1;
    """

    qt_filter_select5 """
         select * from internal.filter_alias_test.test b where internal.filter_alias_test.b.id = 1;
    """
 2024-08-22 14:36:14 +08:00
morrySnow
83acdc14f4 [fix](Nereids) should push project through limit after eliminate union node (#39640) (#39755) 
pick from master #39640

otherwise:

push limit through union could generate plan:

limit
+-- union
    |-- limit
    |   +-- empty relation
    +-- limit
        +-- project

and then eliminate union will generate plan:

+-- limit
    +-  project
        +-- limit
            +-- project

it could not be processed by tranlator correctly
 2024-08-22 13:57:39 +08:00
xzj7019
8f580b523f [opt](nereids) support partitionTopn for multi window exprs (#39687) 
## Proposed changes

pick from https://github.com/apache/doris/pull/38393

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-22 10:34:36 +08:00
seawinde
021982fc71 [fix](mtmv) Fix some pr to 21, prs are (#39041)(#38958)(#39541) (#39678) 
## Proposed changes

pr: https://github.com/apache/doris/pull/39041
commitId: 22562985

pr: https://github.com/apache/doris/pull/38958
commitId: c365cb64

pr: https://github.com/apache/doris/pull/39541
commitId: 89bb669c
 2024-08-22 10:27:55 +08:00
morrySnow
92671cbb73 [opt](Nereids) do not fallback if nereids failed because timeout (#39499) (#39718) 
pick from master #39499

since legacy planner will cost more time to plan, fallback will be worse
than throw exception directly
 2024-08-22 00:45:23 +08:00
wangbo
a55e109e97 [pick][Improment]Add schema table workload_group_privileges (#38436) (#39708) 
pick #38436
 2024-08-22 00:44:43 +08:00
Vallish Pai
935d0eb110 [cherry-pick](branch-2.1) [Bug](http-api) fix core dump on API check_rpc_channel coz exec_env not initialized #39519 (#39692) 
…rpc_channel coz exec_env not initialized #39519

## Proposed changes

Issue Number: close #xxx
backport #39519  #39520
 2024-08-22 00:42:12 +08:00
Tiewei Fang
a613627a00 [fix](test) fix p2 regression test of export (#39683) 
bp: #39088
 2024-08-22 00:41:43 +08:00
Pxl
63d45f5d89 [Bug](predicate) fix wrong result of AcceptNullPredicate (#39497) (#39672) 
pick from #39497
 2024-08-22 00:24:57 +08:00
Mryange
e51dd68b93 [fix](local shuffle) Fix correctness for bucket hash shuffle exchange… (#39691) 
…r (#39568)

For query plan


![image](https://github.com/user-attachments/assets/334cc4c4-49ae-4330-83ff-03b9bae00e3c)

we will plan local exchangers  and get a new plan


![image](https://github.com/user-attachments/assets/2b8ece64-3aa0-423c-9db0-fd02024957db)

and the hash join operator will get probe and build data which are
different distributed (one is HASH shuffle and another is Bucket hash
shuffle). This PR fix it.
<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
 2024-08-22 00:23:39 +08:00
slothever
ebbebdf590 [regression](kerberos)add hive with kerberos write back case (#39682) 
bp #38647
 2024-08-21 18:29:42 +08:00
meiyi
1460878bdf [fix](cluster key) forbid cluster key and remove case (#39679) 
branch-2.1 does not support mow cluster key
 2024-08-21 14:31:54 +08:00
starocean999
ba3b56d269 [fix](nereids)prevent null pointer exception if datetime value overflows (#39675) 
pick from master https://github.com/apache/doris/pull/39482
 2024-08-21 14:17:34 +08:00
Mingyu Chen
0bfcee1251 [opt](file-cache) support system table file_cache_statistics (#39552) 
1. Add new system table: `file_cache_statistics`

	This table is used for viewing metrics related to file cache on BE side

	```
	mysql> select * from information_schema.file_cache_statistics limit 10;

+-------+---------------+----------------------------+--------------------------------+--------------------+
| BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE |

+-------+---------------+----------------------------+--------------------------------+--------------------+
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_elements | 102400 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_size | 21474836480 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio |
0.8539634687001242 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_max_elements | 102400 |

+-------+---------------+----------------------------+--------------------------------+--------------------+
	```

	It will show metrics of file caches on each BE.

2. Add new metrics `hits_ratio_1h` and `hits_ratio_5m` for file cache

This 2 metrics will show the hit ratio of file cache in recent 1 hour or
5 minutes.
So that we can know recent hit ratio instead of global historical hit
ratio.
 2024-08-21 10:03:39 +08:00
wuwenchi
bf26f49505 [bugfix](external)add check of engine and catalog types for 2.1 #39343 (#39643) 
bp #39343
 2024-08-21 09:50:17 +08:00
xzj7019
8a562aeb77 [opt](nereids) recover adoptive bucket shuffle (#39598) 
## Proposed changes

pick from https://github.com/apache/doris/pull/36784

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-21 09:26:53 +08:00
wuwenchi
6df6f1dc97 [improvement](iceberg)]support doris's char/varchar to iceberg's string for 2.1 #38807 (#39645) 
bp: #38807
 2024-08-21 09:19:10 +08:00
wuwenchi
28ce116b17 [improvement](iceberg)add some description for show create for 2.1 #39179 (#39644) 
## Proposed changes

bp: #39179

1. add `location` and `properties` for `show create table`.
2. add `location` for `show create database`.
 2024-08-21 09:18:38 +08:00
daidai
27ba2542e2 [case](iceberg)append iceberg schema change case. (#38766) (#39630) 
bp #38766
 2024-08-21 09:17:12 +08:00