database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,911 Commits 1 Branch 0 Tags
6ceb574aa0c2dc123e259ea42df72b79c563ea0f
Commit Graph

19911 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
wangbo
6ceb574aa0 [branch-2.1]Pick IO limit/workload group usage table (#39839) 2024-08-23 18:51:47 +08:00
Xinyi Zou
e716658fba [branch-2.1](arrow-flight-sql) Fix exceed user property max connection cause Reach limit of connections (#39836) 
pick #39127
pick #39802
 2024-08-23 17:27:34 +08:00
xzj7019
424ad2384a [opt](nereids) refine left semi/anti cost under short-cut opt (#39636) 
## Proposed changes

pick from https://github.com/apache/doris/pull/37951

---------

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-23 17:26:56 +08:00
Xinyi Zou
baf5b71b39 [branch-2.1](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache (#39829) 
pick #38185
 2024-08-23 17:21:42 +08:00
wuwenchi
c40246efa9 [bugfix](iceberg)Fixed random core with writing iceberg partitioned table for 2.1 (#39808)(#39569) (#39832) 
## Proposed changes

bp: #39808 #39569
 2024-08-23 17:19:48 +08:00
Xujian Duan
8f15efdbb8 [cherry-pick](branch-2.1) fix delete random distributed tbl (#39830) 
## Proposed changes

cherry-pick #37985

<!--Describe your changes.-->
 2024-08-23 17:17:05 +08:00
wangbo
7a7292ad5a [branch-2.1][Refactor]use async to get be resource (#38389) (#39826) 
pick #38389
 2024-08-23 17:16:19 +08:00
Mryange
e03b887a97 [opt](MultiCast) Avoid copying while holding a lock (#37462) (#39816) 
Previously, copying was done while holding a lock; Now, get block while
holding the lock and then copy
https://github.com/apache/doris/pull/37462
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 17:15:34 +08:00
slothever
67a8099991 [fix](multi-catalog)fix max compute array and map type read offset (#39822) 
bp #39680
 2024-08-23 16:53:52 +08:00
walter
e0b65d0ff4 [fix](regression) Fix creating db for downstream url #39601 (#39818) 
cherry pick from #39601
 2024-08-23 16:15:19 +08:00
abmdocrt
0eadfbefc6 [Fix](branch-2.1) Fix wrong thrift index introduced by #37830 (#39824) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 15:05:35 +08:00
yujun
8bbd3db1bc [branch-2.1](thrift) fix TLoadTxnBeginRequest backend_id's field no (#39823) 
Make backend_id's field no the same with master branch.

For upgrading, change backend_id's field no is safe, because old fe can
torrent with TLoadTxnBeginRequest not setting backend id.

backend_id was introduce by #36437
 2024-08-23 12:24:55 +08:00
yujun
0934fbee7e [improvement](query) prefer to chose tablet on alive disk #39467 (#39654) 
cherry pick from #39467
 2024-08-23 12:23:12 +08:00
Mingyu Chen
1f16daa5f6 Revert "[bugfix](iceberg)clear block for partition values for 2.1 (#39569)" (#39815) 
Reverts apache/doris#39729
 2024-08-23 11:58:42 +08:00
htyoung
6c10c47f79 [fix](fe) LIST partition table support modify default bucket num (#39688) 
## Proposed changes
bp #39696

Issue Number: close #39684

```sql
CREATE TABLE `test1` (
    `id1` VARCHAR(255) NULL COMMENT 'id1',
    `id2` VARCHAR(255) NULL COMMENT 'id2',
    `event_time` VARCHAR(255) NULL COMMENT '事件时间',
    `event_date` VARCHAR(255) NULL COMMENT '事件日期',
    `event_ts` VARCHAR(256) NULL COMMENT '事件发生时间戳(毫秒)',
    `dt` VARCHAR(255) NOT NULL COMMENT '日期分区',
    `hr` VARCHAR(255) NOT NULL COMMENT '小时分区'
  ) ENGINE = OLAP DUPLICATE KEY(`id1`) COMMENT 'xxx' PARTITION BY LIST(`dt`, `hr`) (
    PARTITION p2024082021 VALUES IN (("2024-08-20", "21"))
  ) DISTRIBUTED BY HASH(`dt`, `hr`) BUCKETS 2 PROPERTIES (
    "replication_allocation" = "tag.location.default: 1",
    "min_load_replica_num" = "-1",
    "is_being_synced" = "false",
    "storage_medium" = "hdd",
    "storage_format" = "V2",
    "inverted_index_storage_format" = "V1",
    "light_schema_change" = "true",
    "disable_auto_compaction" = "false",
    "enable_single_replica_compaction" = "false",
    "group_commit_interval_ms" = "10000",
    "group_commit_data_bytes" = "134217728"
  );
```

1. 修改前表已有分区的bucket num为2

![image](https://github.com/user-attachments/assets/77efdd0c-f845-41a4-9a31-e454808ffe67)

2. 修改List分区表bucket num（从2 -> 4）

![image](https://github.com/user-attachments/assets/53b19918-2879-4cb3-b2bd-84ba35a7fc59)

3. 修改List分区后，添加的新分区bucket num为4

![image](https://github.com/user-attachments/assets/1f41f73f-d70f-433e-a7b6-8346b7dfcc4e)

Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>
 2024-08-23 11:52:16 +08:00
zy-kkk
40a58b9e42 [branch-2.1][regression test](jdbc catalog) Enable CLICKHOUSE_ALWAYS_RUN_INITDB_SCRIPTS for clickhouse docker (#39667) 
pick (#39425) #39693
 2024-08-23 09:59:03 +08:00
Xinyi Zou
9d5468d198 [branch-2.1](memory) BE memory info compatible with CgroupV2 (#39799) 
pick #39256
 2024-08-23 02:03:00 +08:00
Xinyi Zou
1367f74e7a [branch-2.1](memory) Optimize ClearCacheActionimplementation (#39796) 
pick #38438
 2024-08-23 01:51:14 +08:00
Mryange
0f8bd33077 [fix](scan) fix predicate contains cast that results in null, the pr… (#39809) 
…edicate will be miss. (#39550)
https://github.com/apache/doris/pull/39550
```
drop table datetest;

create table datetest (
  id int,
  dt date
)
DUPLICATE key (id)
distributed by hash(id) buckets 1
properties(
  "replication_num" = "1"
);
insert into datetest values (1, '2024-01-01');

mysql [test10]>select dt from datetest  WHERE dt = 1 ;
+------------+
| dt         |
+------------+
| 2024-01-01 |
+------------+
```

now

```
mysql [test10]>select dt from datetest  WHERE dt = 1 ;
Empty set (0.16 sec)
```

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 01:46:22 +08:00
wuwenchi
dc732fe33f [bugfix](iceberg)clear block for partition values for 2.1 (#39569) (#39729) 
## Proposed changes

bp: #39569

clear block, or we will get wrong partition values.
 2024-08-22 22:43:02 +08:00
Socrates
eeacca9e8e [cherry-pick](branch-2.1) fix hudi count bug (#39785) 
bp #39656
 2024-08-22 22:35:44 +08:00
zhiqiang
06a0b35704 [chore] Comment for tv_nsec (#39752) 
just comment.
 2024-08-22 22:16:21 +08:00
Mryange
04e993c1de [refine](pipeline) refine some VDataStreamRecvr code (#35063) (#37802) 
## Proposed changes
https://github.com/apache/doris/pull/35063
https://github.com/apache/doris/pull/35428
 2024-08-22 19:55:17 +08:00
morrySnow
29898d0dbc [fix](downgrade) partition key should write upper case null type to image (#39776) 2024-08-22 18:42:08 +08:00
Xinyi Zou
40500f2b10 [branch-2.1](thirdparty) upgrade arrow to 17.0.0 (#39773) 
pick #38572
 2024-08-22 18:35:32 +08:00
Xinyi Zou
13b882a4cc [branch-2.1](memory) Add memory metrics to bvar (#39763) 
pick #38391
 2024-08-22 17:34:30 +08:00
Xinyi Zou
8ce8887b75 [branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth (#39760) 
pick #38168
overwrites changes in #37221 on workload_group_manager.cpp. If need to
pick 37221, ignore it.
 2024-08-22 17:33:11 +08:00
TengJianPing
ba7baa7e6b [fix](window_funnel) fix upgrading problem caused by behaviour change of window_funnel (#39766) 
## Proposed changes

Issue Number: close #xxx

For the latest published 2.1 version `2.1.5`:
```
max_be_exec_version=4;
AGG_FUNCTION_NEW=2;
```
and `branch-2.1`:
```
max_be_exec_version=5;
AGG_FUNCTION_NEW=2;
```
It will cause problem when upgrading.
This PR fix the problem, set `AGG_FUNCTION_NEW` to `5`.
 2024-08-22 17:26:51 +08:00
zhangdong
f553645a71 [fix](mtmv) transfer col in mysql varchar to text when create MTMV (#37668) (#39727) 
pick from master #37668
 2024-08-22 15:20:59 +08:00
xzj7019
10f3e88f7a [fix](nereids) fix distribution expr list (#39435) 
pick from #39148
 2024-08-22 15:19:51 +08:00
wangbo
1c566253a8 [Pick][Improment]Query queued by be memory (#37559) (#39733) 
pick #37559
 2024-08-22 15:14:47 +08:00
toms
fd13962015 [chore](nereids) Added compatibility with mysql alias conflict (#38104) (#38440) 
throw table name/alias conflict exception to keep same behavior with mysql

for example:
```sql
select * from test.a b, test.b
```

error:
```
Not unique table/alias: 'b'
```
 2024-08-22 14:37:49 +08:00
toms
50f440e653 [chore](nereids) Added compatibility with mysql alias filter (#39738) 
qt_filter_select4 """
       select * from filter_alias_test.test b where filter_alias_test.b.id = 1;
    """

    qt_filter_select5 """
         select * from internal.filter_alias_test.test b where internal.filter_alias_test.b.id = 1;
    """
 2024-08-22 14:36:14 +08:00
morrySnow
83acdc14f4 [fix](Nereids) should push project through limit after eliminate union node (#39640) (#39755) 
pick from master #39640

otherwise:

push limit through union could generate plan:

limit
+-- union
    |-- limit
    |   +-- empty relation
    +-- limit
        +-- project

and then eliminate union will generate plan:

+-- limit
    +-  project
        +-- limit
            +-- project

it could not be processed by tranlator correctly
 2024-08-22 13:57:39 +08:00
LiBinfeng
ca9e50e49d [fix](Nereids) fix insert into table with null literal default value (#39122) (#39669) 
cherry-pick: #39122

Problem:
when use insert with default value null, it can not be insert
successfully
Solved:
when column is allow to be null, it can be null in create table with
null default value

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-22 10:37:50 +08:00
xzj7019
8f580b523f [opt](nereids) support partitionTopn for multi window exprs (#39687) 
## Proposed changes

pick from https://github.com/apache/doris/pull/38393

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-22 10:34:36 +08:00
seawinde
021982fc71 [fix](mtmv) Fix some pr to 21, prs are (#39041)(#38958)(#39541) (#39678) 
## Proposed changes

pr: https://github.com/apache/doris/pull/39041
commitId: 22562985

pr: https://github.com/apache/doris/pull/38958
commitId: c365cb64

pr: https://github.com/apache/doris/pull/39541
commitId: 89bb669c
 2024-08-22 10:27:55 +08:00
zhengyu
d36ba97eae [fix](statistics) correct update rows when doing multi-table load (#39548) 
rows of only one table is updated correctly, need to merge all table
commit infos.
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
 2024-08-22 09:57:45 +08:00
wangbo
ed9794a0fe [Pick][Improment]publish workload to BE by tag (#38486) (#39730) 
A workload group's tag property may be three cases as below: 1 empty
string, null or '', it could be published to all BE. 2 a value match
some BE' location, then the workload group could only be published to
the BE with same tag.
3 not an empty string, but some invalid string which can not math any
BE's location, then it could not be published any BE.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-22 00:48:16 +08:00
morrySnow
92671cbb73 [opt](Nereids) do not fallback if nereids failed because timeout (#39499) (#39718) 
pick from master #39499

since legacy planner will cost more time to plan, fallback will be worse
than throw exception directly
 2024-08-22 00:45:23 +08:00
wangbo
a55e109e97 [pick][Improment]Add schema table workload_group_privileges (#38436) (#39708) 
pick #38436
 2024-08-22 00:44:43 +08:00
zhannngchen
0e694f19db [fix](merge-on-write) segcompaction should process delete bitmap if necessary (#38369) (#39707) 
## Proposed changes

Issue Number: close #xxx

cherry-pick #38369 and #38800
 2024-08-22 00:42:56 +08:00
Vallish Pai
935d0eb110 [cherry-pick](branch-2.1) [Bug](http-api) fix core dump on API check_rpc_channel coz exec_env not initialized #39519 (#39692) 
…rpc_channel coz exec_env not initialized #39519

## Proposed changes

Issue Number: close #xxx
backport #39519  #39520
 2024-08-22 00:42:12 +08:00
Tiewei Fang
a613627a00 [fix](test) fix p2 regression test of export (#39683) 
bp: #39088
 2024-08-22 00:41:43 +08:00
zhiqiang
56cc9cc304 [fix](cancel)) Fix pipeline task leak cancel (#39697) 
pick #39737
 2024-08-22 00:40:22 +08:00
Pxl
1e47d11560 [Improvement](runtime-filter) send RUNTIME_BLOOM_FILTER_MAX_SIZE to backends (#39686) 
…ackends (#38972)

## Proposed changes
pick from #38972
 2024-08-22 00:37:25 +08:00
camby
b5bdc325af [fix](profile) task type not the same in observer and master (#39245) (#39698) 
pick #39245 to branch-2.1
 2024-08-22 00:31:52 +08:00
Pxl
5e91fc6a8f [Bug](runtime-filter) set inited to true on BloomFilterFuncBase::assi… (#39674) 
…gn (#39335)

## Proposed changes
pick from #39335
 2024-08-22 00:29:16 +08:00
Pxl
63d45f5d89 [Bug](predicate) fix wrong result of AcceptNullPredicate (#39497) (#39672) 
pick from #39497
 2024-08-22 00:24:57 +08:00
Mryange
e51dd68b93 [fix](local shuffle) Fix correctness for bucket hash shuffle exchange… (#39691) 
…r (#39568)

For query plan


![image](https://github.com/user-attachments/assets/334cc4c4-49ae-4330-83ff-03b9bae00e3c)

we will plan local exchangers  and get a new plan


![image](https://github.com/user-attachments/assets/2b8ece64-3aa0-423c-9db0-fd02024957db)

and the hash join operator will get probe and build data which are
different distributed (one is HASH shuffle and another is Bucket hash
shuffle). This PR fix it.
<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
 2024-08-22 00:23:39 +08:00