database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,925 Commits 1 Branch 0 Tags
de2e8f0ae6ab0d60830701366e4e95cf5895ad66
Commit Graph

19925 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mingyu Chen
de2e8f0ae6 [fix](ctas) fix NPE when ctas with old planner and varchar issue (#39744) (#39871) 
bp #39744
 2024-08-24 09:24:47 +08:00
Mingyu Chen
b9da934b16 [fix](hive) report error with escape char and null format (#39700) (#39869) 
bp #39700

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-08-24 09:23:03 +08:00
lihangyu
a6f267c479 [pick](Variant) fix element_at should return nullable if result type is nullable (#39846) 
#39732
 2024-08-24 09:22:03 +08:00
Xinyi Zou
ae4d747c13 [branch-2.1](memory) Modify memory gc conf and add crash_in_alloc_large_memory_bytes (#39834) 
pick #39611
 2024-08-24 09:21:35 +08:00
Jerry Hu
8cf6c6a2b5 [fix](agg function) incorrect result of map agg(#39743) (#39854) 
## Proposed changes

pick #39743
 2024-08-24 09:19:52 +08:00
Mingyu Chen
cf698fb615 [fix](regression) fix some jdbc datasource docker health check (#39141) (#39872) 2024-08-24 03:29:18 +08:00
zy-kkk
4cb7122a89 [branch-2.1][fix](jdbc catalog) Fix Memory Leak by Enabling Weak References in HikariCP (#39833) 
pick(#39582)
 2024-08-24 00:45:12 +08:00
Siyang Tang
00e4f343b0 [enhancement](delete) Add a hint msg for forbidden delete when MV or rollup exists (#39505) (#39857) 
## Proposed changes

When MV or Rollup exists, delete is forbidden on the base table
currently. Add a hint msg to indicate it.
 2024-08-23 23:42:26 +08:00
morrySnow
9d597bde68 [fix](Nereids) producer to consumer should be multimap in cte (#39850) (#39867) 2024-08-23 23:25:11 +08:00
Mingyu Chen
508c7a7040 [fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. (#39239) (#39865) 
bp #39239

Co-authored-by: daidai <2017501503@qq.com>
 2024-08-23 23:21:02 +08:00
lihangyu
3103bb08dc [pick](Variant) casting to decimal type may lost precision (#39843) 
#39650
 2024-08-23 22:47:32 +08:00
Kaijie Chen
37443aa7e1 [improve](move-memtable) reuse connection in load_stream_stub (#39231) (#39762) 
backport #39231
 2024-08-23 22:46:28 +08:00
Siyang Tang
5b124a03ba [enhancement](err-msg) Add detailed column and schema info when failed to create a column iterator (#38689) (#39861) 
As title.
 2024-08-23 21:39:19 +08:00
zhangdong
19c51cc217 [fix](mtmv) use name instead of id in meta of MTMV (#39355) (#39748) 
pick from master #39355
 2024-08-23 20:40:54 +08:00
wangbo
6ceb574aa0 [branch-2.1]Pick IO limit/workload group usage table (#39839) 2024-08-23 18:51:47 +08:00
Xinyi Zou
e716658fba [branch-2.1](arrow-flight-sql) Fix exceed user property max connection cause Reach limit of connections (#39836) 
pick #39127
pick #39802
 2024-08-23 17:27:34 +08:00
xzj7019
424ad2384a [opt](nereids) refine left semi/anti cost under short-cut opt (#39636) 
## Proposed changes

pick from https://github.com/apache/doris/pull/37951

---------

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-23 17:26:56 +08:00
Xinyi Zou
baf5b71b39 [branch-2.1](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache (#39829) 
pick #38185
 2024-08-23 17:21:42 +08:00
wuwenchi
c40246efa9 [bugfix](iceberg)Fixed random core with writing iceberg partitioned table for 2.1 (#39808)(#39569) (#39832) 
## Proposed changes

bp: #39808 #39569
 2024-08-23 17:19:48 +08:00
Xujian Duan
8f15efdbb8 [cherry-pick](branch-2.1) fix delete random distributed tbl (#39830) 
## Proposed changes

cherry-pick #37985

<!--Describe your changes.-->
 2024-08-23 17:17:05 +08:00
wangbo
7a7292ad5a [branch-2.1][Refactor]use async to get be resource (#38389) (#39826) 
pick #38389
 2024-08-23 17:16:19 +08:00
Mryange
e03b887a97 [opt](MultiCast) Avoid copying while holding a lock (#37462) (#39816) 
Previously, copying was done while holding a lock; Now, get block while
holding the lock and then copy
https://github.com/apache/doris/pull/37462
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 17:15:34 +08:00
slothever
67a8099991 [fix](multi-catalog)fix max compute array and map type read offset (#39822) 
bp #39680
 2024-08-23 16:53:52 +08:00
walter
e0b65d0ff4 [fix](regression) Fix creating db for downstream url #39601 (#39818) 
cherry pick from #39601
 2024-08-23 16:15:19 +08:00
abmdocrt
0eadfbefc6 [Fix](branch-2.1) Fix wrong thrift index introduced by #37830 (#39824) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 15:05:35 +08:00
yujun
8bbd3db1bc [branch-2.1](thrift) fix TLoadTxnBeginRequest backend_id's field no (#39823) 
Make backend_id's field no the same with master branch.

For upgrading, change backend_id's field no is safe, because old fe can
torrent with TLoadTxnBeginRequest not setting backend id.

backend_id was introduce by #36437
 2024-08-23 12:24:55 +08:00
yujun
0934fbee7e [improvement](query) prefer to chose tablet on alive disk #39467 (#39654) 
cherry pick from #39467
 2024-08-23 12:23:12 +08:00
Mingyu Chen
1f16daa5f6 Revert "[bugfix](iceberg)clear block for partition values for 2.1 (#39569)" (#39815) 
Reverts apache/doris#39729
 2024-08-23 11:58:42 +08:00
htyoung
6c10c47f79 [fix](fe) LIST partition table support modify default bucket num (#39688) 
## Proposed changes
bp #39696

Issue Number: close #39684

```sql
CREATE TABLE `test1` (
    `id1` VARCHAR(255) NULL COMMENT 'id1',
    `id2` VARCHAR(255) NULL COMMENT 'id2',
    `event_time` VARCHAR(255) NULL COMMENT '事件时间',
    `event_date` VARCHAR(255) NULL COMMENT '事件日期',
    `event_ts` VARCHAR(256) NULL COMMENT '事件发生时间戳(毫秒)',
    `dt` VARCHAR(255) NOT NULL COMMENT '日期分区',
    `hr` VARCHAR(255) NOT NULL COMMENT '小时分区'
  ) ENGINE = OLAP DUPLICATE KEY(`id1`) COMMENT 'xxx' PARTITION BY LIST(`dt`, `hr`) (
    PARTITION p2024082021 VALUES IN (("2024-08-20", "21"))
  ) DISTRIBUTED BY HASH(`dt`, `hr`) BUCKETS 2 PROPERTIES (
    "replication_allocation" = "tag.location.default: 1",
    "min_load_replica_num" = "-1",
    "is_being_synced" = "false",
    "storage_medium" = "hdd",
    "storage_format" = "V2",
    "inverted_index_storage_format" = "V1",
    "light_schema_change" = "true",
    "disable_auto_compaction" = "false",
    "enable_single_replica_compaction" = "false",
    "group_commit_interval_ms" = "10000",
    "group_commit_data_bytes" = "134217728"
  );
```

1. 修改前表已有分区的bucket num为2

![image](https://github.com/user-attachments/assets/77efdd0c-f845-41a4-9a31-e454808ffe67)

2. 修改List分区表bucket num（从2 -> 4）

![image](https://github.com/user-attachments/assets/53b19918-2879-4cb3-b2bd-84ba35a7fc59)

3. 修改List分区后，添加的新分区bucket num为4

![image](https://github.com/user-attachments/assets/1f41f73f-d70f-433e-a7b6-8346b7dfcc4e)

Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>
 2024-08-23 11:52:16 +08:00
zy-kkk
40a58b9e42 [branch-2.1][regression test](jdbc catalog) Enable CLICKHOUSE_ALWAYS_RUN_INITDB_SCRIPTS for clickhouse docker (#39667) 
pick (#39425) #39693
 2024-08-23 09:59:03 +08:00
Xinyi Zou
9d5468d198 [branch-2.1](memory) BE memory info compatible with CgroupV2 (#39799) 
pick #39256
 2024-08-23 02:03:00 +08:00
Xinyi Zou
1367f74e7a [branch-2.1](memory) Optimize ClearCacheActionimplementation (#39796) 
pick #38438
 2024-08-23 01:51:14 +08:00
Mryange
0f8bd33077 [fix](scan) fix predicate contains cast that results in null, the pr… (#39809) 
…edicate will be miss. (#39550)
https://github.com/apache/doris/pull/39550
```
drop table datetest;

create table datetest (
  id int,
  dt date
)
DUPLICATE key (id)
distributed by hash(id) buckets 1
properties(
  "replication_num" = "1"
);
insert into datetest values (1, '2024-01-01');

mysql [test10]>select dt from datetest  WHERE dt = 1 ;
+------------+
| dt         |
+------------+
| 2024-01-01 |
+------------+
```

now

```
mysql [test10]>select dt from datetest  WHERE dt = 1 ;
Empty set (0.16 sec)
```

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 01:46:22 +08:00
wuwenchi
dc732fe33f [bugfix](iceberg)clear block for partition values for 2.1 (#39569) (#39729) 
## Proposed changes

bp: #39569

clear block, or we will get wrong partition values.
 2024-08-22 22:43:02 +08:00
Socrates
eeacca9e8e [cherry-pick](branch-2.1) fix hudi count bug (#39785) 
bp #39656
 2024-08-22 22:35:44 +08:00
zhiqiang
06a0b35704 [chore] Comment for tv_nsec (#39752) 
just comment.
 2024-08-22 22:16:21 +08:00
Mryange
04e993c1de [refine](pipeline) refine some VDataStreamRecvr code (#35063) (#37802) 
## Proposed changes
https://github.com/apache/doris/pull/35063
https://github.com/apache/doris/pull/35428
 2024-08-22 19:55:17 +08:00
morrySnow
29898d0dbc [fix](downgrade) partition key should write upper case null type to image (#39776) 2024-08-22 18:42:08 +08:00
Xinyi Zou
40500f2b10 [branch-2.1](thirdparty) upgrade arrow to 17.0.0 (#39773) 
pick #38572
 2024-08-22 18:35:32 +08:00
Xinyi Zou
13b882a4cc [branch-2.1](memory) Add memory metrics to bvar (#39763) 
pick #38391
 2024-08-22 17:34:30 +08:00
Xinyi Zou
8ce8887b75 [branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth (#39760) 
pick #38168
overwrites changes in #37221 on workload_group_manager.cpp. If need to
pick 37221, ignore it.
 2024-08-22 17:33:11 +08:00
TengJianPing
ba7baa7e6b [fix](window_funnel) fix upgrading problem caused by behaviour change of window_funnel (#39766) 
## Proposed changes

Issue Number: close #xxx

For the latest published 2.1 version `2.1.5`:
```
max_be_exec_version=4;
AGG_FUNCTION_NEW=2;
```
and `branch-2.1`:
```
max_be_exec_version=5;
AGG_FUNCTION_NEW=2;
```
It will cause problem when upgrading.
This PR fix the problem, set `AGG_FUNCTION_NEW` to `5`.
 2024-08-22 17:26:51 +08:00
zhangdong
f553645a71 [fix](mtmv) transfer col in mysql varchar to text when create MTMV (#37668) (#39727) 
pick from master #37668
 2024-08-22 15:20:59 +08:00
xzj7019
10f3e88f7a [fix](nereids) fix distribution expr list (#39435) 
pick from #39148
 2024-08-22 15:19:51 +08:00
wangbo
1c566253a8 [Pick][Improment]Query queued by be memory (#37559) (#39733) 
pick #37559
 2024-08-22 15:14:47 +08:00
toms
fd13962015 [chore](nereids) Added compatibility with mysql alias conflict (#38104) (#38440) 
throw table name/alias conflict exception to keep same behavior with mysql

for example:
```sql
select * from test.a b, test.b
```

error:
```
Not unique table/alias: 'b'
```
 2024-08-22 14:37:49 +08:00
toms
50f440e653 [chore](nereids) Added compatibility with mysql alias filter (#39738) 
qt_filter_select4 """
       select * from filter_alias_test.test b where filter_alias_test.b.id = 1;
    """

    qt_filter_select5 """
         select * from internal.filter_alias_test.test b where internal.filter_alias_test.b.id = 1;
    """
 2024-08-22 14:36:14 +08:00
morrySnow
83acdc14f4 [fix](Nereids) should push project through limit after eliminate union node (#39640) (#39755) 
pick from master #39640

otherwise:

push limit through union could generate plan:

limit
+-- union
    |-- limit
    |   +-- empty relation
    +-- limit
        +-- project

and then eliminate union will generate plan:

+-- limit
    +-  project
        +-- limit
            +-- project

it could not be processed by tranlator correctly
 2024-08-22 13:57:39 +08:00
LiBinfeng
ca9e50e49d [fix](Nereids) fix insert into table with null literal default value (#39122) (#39669) 
cherry-pick: #39122

Problem:
when use insert with default value null, it can not be insert
successfully
Solved:
when column is allow to be null, it can be null in create table with
null default value

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-22 10:37:50 +08:00
xzj7019
8f580b523f [opt](nereids) support partitionTopn for multi window exprs (#39687) 
## Proposed changes

pick from https://github.com/apache/doris/pull/38393

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-22 10:34:36 +08:00