database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,939 Commits 1 Branch 0 Tags
44b80fb03b282eecd9740a7cc00a94847f683c2b
Commit Graph

19939 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zzzxl
44b80fb03b [fix](inverted index) Fix Session Variable Compatibility (#39884) 
https://github.com/apache/doris/pull/39889
 2024-08-25 08:42:36 +08:00
TengJianPing
8dbd73988a [fix](recvr) catch exception of transmit_block (#39882) 
BP #39881
 2024-08-25 00:25:20 +08:00
yiguolei
e0534c9bfc [bugfix](thrift) the definition number should consistent with master branch (#39879) 
## Proposed changes
introduced by pr https://github.com/apache/doris/pull/35103

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-08-25 00:22:19 +08:00
LiBinfeng
9997911ec9 [feat](Nereids) support nereids hint position detaction (#39113) (#39417) 
cherry-pick: #39113
When use hint in wrong position or use unsupport hint, use channel(2) to
filter it out

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-24 23:59:54 +08:00
Gavin Chou
8e140727ae [fix](udf) Fix exception thrown when replayDropFunction (#39868) 
This PR completes #25965 to fix uncaught runtime exception when calling
`org.apache.doris.catalog.GlobalFunctionMgr#replayDropFunction`


![img_v3_02e0_3889b1de-583e-48ba-8e7f-48e1d3ae799g](https://github.com/user-attachments/assets/b0dbfefb-16f3-4dbb-8a00-f4773978ba01)
 2024-08-24 23:56:48 +08:00
TengJianPing
2dea859bdb [debug](rpc) debug rpc time consumption problem (#39852) 
## Proposed changes

Issue Number: close #xxx

Add detail RPC time info for each channel, sorted by max rpc time of
channels:
```
                     DATA_STREAM_SINK_OPERATOR  (id=1,dst_id=1):
                          -  Partitioner:  Crc32HashPartitioner(64)
                          -  BlocksProduced:  74
                          -  BrpcSendTime:  2.689us
                          -  BrpcSendTime.Wait:  0ns
                          -  BytesSent:  89.35  KB
                          -  CloseTime:  680.152us
                          -  CompressTime:  0ns
                          -  ExecTime:  160.663ms
                          -  InitTime:  263.608us
                          -  InputRows:  32.512K  (32512)
                          -  LocalBytesSent:  0.00  
                          -  LocalSendTime:  0ns
                          -  LocalSentRows:  0
                          -  MemoryUsage:  
                              -  PeakMemoryUsage:  80.00  KB
                          -  MergeBlockTime:  0ns
                          -  OpenTime:  4.113ms
                          -  OverallThroughput:  0.0  /sec
                          -  PendingFinishDependency:  41.179ms
                          -  RowsProduced:  32.512K  (32512)
                          -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m
```
 2024-08-24 19:59:39 +08:00
Jerry Hu
5a810122a2 [debug](load) check the column type when string column is invalid (#39337) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-24 18:14:21 +08:00
Mingyu Chen
263746b04b [fix](paimon) fix crash when enable cache with paimon deletion vector(#39877) (#39875) 
bp #39877
 2024-08-24 17:58:20 +08:00
Mingyu Chen
14a2a66106 [fix](paimon) fix not able to read paimon data from hdfs with HA (#39806) (#39876) 
bp #39806
 2024-08-24 17:51:15 +08:00
zclllhhjj
460605ae3c [branch-2.1] pick some prs (#39860) 
## Proposed changes

Issue Number: close #xxx

https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
 2024-08-24 17:26:42 +08:00
HappenLee
564d3cd647 [Performance](opt) opt the order by performance in permutation (#39089) 
## Proposed changes

Issue Number: cherry pick #38985

<!--Describe your changes.-->
 2024-08-24 16:05:46 +08:00
Mingyu Chen
9640e2de44 [fix](catalog) refactor location path and support default fs #39116 (#39203) 2024-08-24 16:05:13 +08:00
Kang
76596e5f73 [fix](thrift) fix wrong order of field 27 and 28 in TCreateTabletReq thrift (#39873) 
## Proposed changes

fix wrong order of field 27 and 28 in `TCreateTabletReq` thrift
introduced by #32418 and 0cde0cbf19011bc8d421add4734d7cd57308973f.

`TCreateTabletReq` is used for creating tablet, so this PR will fix
problem creating tablet when upgrading 2.0.x -> 2.1.4/2.1.5 or
2.1.4/2.1.5 -> 3.0.x, BUT will cause problem creating tablet when
upgrading 2.1.4/2.1.5 -> 2.1.6+.

master and branch-2.0 
```
    27: optional i64 time_series_compaction_level_threshold = 1
    28: optional TInvertedIndexStorageFormat inverted_index_storage_format = TInvertedIndexStorageFormat.V1
```

branch-2.1 (affects 2.1.4 and 2.1.5)
```
    27: optional TInvertedIndexStorageFormat inverted_index_storage_format = TInvertedIndexStorageFormat.V1
    28: optional i64 time_series_compaction_level_threshold = 1
```
 2024-08-24 16:02:08 +08:00
Mingyu Chen
5ed56770d4 [bugfix](external) Prevent multiple fs from being generated (#39663) (#39870) 
bp #39663

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
 2024-08-24 14:17:26 +08:00
Mingyu Chen
de2e8f0ae6 [fix](ctas) fix NPE when ctas with old planner and varchar issue (#39744) (#39871) 
bp #39744
 2024-08-24 09:24:47 +08:00
Mingyu Chen
b9da934b16 [fix](hive) report error with escape char and null format (#39700) (#39869) 
bp #39700

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-08-24 09:23:03 +08:00
lihangyu
a6f267c479 [pick](Variant) fix element_at should return nullable if result type is nullable (#39846) 
#39732
 2024-08-24 09:22:03 +08:00
Xinyi Zou
ae4d747c13 [branch-2.1](memory) Modify memory gc conf and add crash_in_alloc_large_memory_bytes (#39834) 
pick #39611
 2024-08-24 09:21:35 +08:00
Jerry Hu
8cf6c6a2b5 [fix](agg function) incorrect result of map agg(#39743) (#39854) 
## Proposed changes

pick #39743
 2024-08-24 09:19:52 +08:00
Mingyu Chen
cf698fb615 [fix](regression) fix some jdbc datasource docker health check (#39141) (#39872) 2024-08-24 03:29:18 +08:00
zy-kkk
4cb7122a89 [branch-2.1][fix](jdbc catalog) Fix Memory Leak by Enabling Weak References in HikariCP (#39833) 
pick(#39582)
 2024-08-24 00:45:12 +08:00
Siyang Tang
00e4f343b0 [enhancement](delete) Add a hint msg for forbidden delete when MV or rollup exists (#39505) (#39857) 
## Proposed changes

When MV or Rollup exists, delete is forbidden on the base table
currently. Add a hint msg to indicate it.
 2024-08-23 23:42:26 +08:00
morrySnow
9d597bde68 [fix](Nereids) producer to consumer should be multimap in cte (#39850) (#39867) 2024-08-23 23:25:11 +08:00
Mingyu Chen
508c7a7040 [fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. (#39239) (#39865) 
bp #39239

Co-authored-by: daidai <2017501503@qq.com>
 2024-08-23 23:21:02 +08:00
lihangyu
3103bb08dc [pick](Variant) casting to decimal type may lost precision (#39843) 
#39650
 2024-08-23 22:47:32 +08:00
Kaijie Chen
37443aa7e1 [improve](move-memtable) reuse connection in load_stream_stub (#39231) (#39762) 
backport #39231
 2024-08-23 22:46:28 +08:00
Siyang Tang
5b124a03ba [enhancement](err-msg) Add detailed column and schema info when failed to create a column iterator (#38689) (#39861) 
As title.
 2024-08-23 21:39:19 +08:00
zhangdong
19c51cc217 [fix](mtmv) use name instead of id in meta of MTMV (#39355) (#39748) 
pick from master #39355
 2024-08-23 20:40:54 +08:00
wangbo
6ceb574aa0 [branch-2.1]Pick IO limit/workload group usage table (#39839) 2024-08-23 18:51:47 +08:00
Xinyi Zou
e716658fba [branch-2.1](arrow-flight-sql) Fix exceed user property max connection cause Reach limit of connections (#39836) 
pick #39127
pick #39802
 2024-08-23 17:27:34 +08:00
xzj7019
424ad2384a [opt](nereids) refine left semi/anti cost under short-cut opt (#39636) 
## Proposed changes

pick from https://github.com/apache/doris/pull/37951

---------

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-23 17:26:56 +08:00
Xinyi Zou
baf5b71b39 [branch-2.1](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache (#39829) 
pick #38185
 2024-08-23 17:21:42 +08:00
wuwenchi
c40246efa9 [bugfix](iceberg)Fixed random core with writing iceberg partitioned table for 2.1 (#39808)(#39569) (#39832) 
## Proposed changes

bp: #39808 #39569
 2024-08-23 17:19:48 +08:00
Xujian Duan
8f15efdbb8 [cherry-pick](branch-2.1) fix delete random distributed tbl (#39830) 
## Proposed changes

cherry-pick #37985

<!--Describe your changes.-->
 2024-08-23 17:17:05 +08:00
wangbo
7a7292ad5a [branch-2.1][Refactor]use async to get be resource (#38389) (#39826) 
pick #38389
 2024-08-23 17:16:19 +08:00
Mryange
e03b887a97 [opt](MultiCast) Avoid copying while holding a lock (#37462) (#39816) 
Previously, copying was done while holding a lock; Now, get block while
holding the lock and then copy
https://github.com/apache/doris/pull/37462
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 17:15:34 +08:00
slothever
67a8099991 [fix](multi-catalog)fix max compute array and map type read offset (#39822) 
bp #39680
 2024-08-23 16:53:52 +08:00
walter
e0b65d0ff4 [fix](regression) Fix creating db for downstream url #39601 (#39818) 
cherry pick from #39601
 2024-08-23 16:15:19 +08:00
abmdocrt
0eadfbefc6 [Fix](branch-2.1) Fix wrong thrift index introduced by #37830 (#39824) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 15:05:35 +08:00
yujun
8bbd3db1bc [branch-2.1](thrift) fix TLoadTxnBeginRequest backend_id's field no (#39823) 
Make backend_id's field no the same with master branch.

For upgrading, change backend_id's field no is safe, because old fe can
torrent with TLoadTxnBeginRequest not setting backend id.

backend_id was introduce by #36437
 2024-08-23 12:24:55 +08:00
yujun
0934fbee7e [improvement](query) prefer to chose tablet on alive disk #39467 (#39654) 
cherry pick from #39467
 2024-08-23 12:23:12 +08:00
Mingyu Chen
1f16daa5f6 Revert "[bugfix](iceberg)clear block for partition values for 2.1 (#39569)" (#39815) 
Reverts apache/doris#39729
 2024-08-23 11:58:42 +08:00
htyoung
6c10c47f79 [fix](fe) LIST partition table support modify default bucket num (#39688) 
## Proposed changes
bp #39696

Issue Number: close #39684

```sql
CREATE TABLE `test1` (
    `id1` VARCHAR(255) NULL COMMENT 'id1',
    `id2` VARCHAR(255) NULL COMMENT 'id2',
    `event_time` VARCHAR(255) NULL COMMENT '事件时间',
    `event_date` VARCHAR(255) NULL COMMENT '事件日期',
    `event_ts` VARCHAR(256) NULL COMMENT '事件发生时间戳(毫秒)',
    `dt` VARCHAR(255) NOT NULL COMMENT '日期分区',
    `hr` VARCHAR(255) NOT NULL COMMENT '小时分区'
  ) ENGINE = OLAP DUPLICATE KEY(`id1`) COMMENT 'xxx' PARTITION BY LIST(`dt`, `hr`) (
    PARTITION p2024082021 VALUES IN (("2024-08-20", "21"))
  ) DISTRIBUTED BY HASH(`dt`, `hr`) BUCKETS 2 PROPERTIES (
    "replication_allocation" = "tag.location.default: 1",
    "min_load_replica_num" = "-1",
    "is_being_synced" = "false",
    "storage_medium" = "hdd",
    "storage_format" = "V2",
    "inverted_index_storage_format" = "V1",
    "light_schema_change" = "true",
    "disable_auto_compaction" = "false",
    "enable_single_replica_compaction" = "false",
    "group_commit_interval_ms" = "10000",
    "group_commit_data_bytes" = "134217728"
  );
```

1. 修改前表已有分区的bucket num为2

![image](https://github.com/user-attachments/assets/77efdd0c-f845-41a4-9a31-e454808ffe67)

2. 修改List分区表bucket num（从2 -> 4）

![image](https://github.com/user-attachments/assets/53b19918-2879-4cb3-b2bd-84ba35a7fc59)

3. 修改List分区后，添加的新分区bucket num为4

![image](https://github.com/user-attachments/assets/1f41f73f-d70f-433e-a7b6-8346b7dfcc4e)

Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>
 2024-08-23 11:52:16 +08:00
zy-kkk
40a58b9e42 [branch-2.1][regression test](jdbc catalog) Enable CLICKHOUSE_ALWAYS_RUN_INITDB_SCRIPTS for clickhouse docker (#39667) 
pick (#39425) #39693
 2024-08-23 09:59:03 +08:00
Xinyi Zou
9d5468d198 [branch-2.1](memory) BE memory info compatible with CgroupV2 (#39799) 
pick #39256
 2024-08-23 02:03:00 +08:00
Xinyi Zou
1367f74e7a [branch-2.1](memory) Optimize ClearCacheActionimplementation (#39796) 
pick #38438
 2024-08-23 01:51:14 +08:00
Mryange
0f8bd33077 [fix](scan) fix predicate contains cast that results in null, the pr… (#39809) 
…edicate will be miss. (#39550)
https://github.com/apache/doris/pull/39550
```
drop table datetest;

create table datetest (
  id int,
  dt date
)
DUPLICATE key (id)
distributed by hash(id) buckets 1
properties(
  "replication_num" = "1"
);
insert into datetest values (1, '2024-01-01');

mysql [test10]>select dt from datetest  WHERE dt = 1 ;
+------------+
| dt         |
+------------+
| 2024-01-01 |
+------------+
```

now

```
mysql [test10]>select dt from datetest  WHERE dt = 1 ;
Empty set (0.16 sec)
```

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-23 01:46:22 +08:00
wuwenchi
dc732fe33f [bugfix](iceberg)clear block for partition values for 2.1 (#39569) (#39729) 
## Proposed changes

bp: #39569

clear block, or we will get wrong partition values.
 2024-08-22 22:43:02 +08:00
Socrates
eeacca9e8e [cherry-pick](branch-2.1) fix hudi count bug (#39785) 
bp #39656
 2024-08-22 22:35:44 +08:00
zhiqiang
06a0b35704 [chore] Comment for tv_nsec (#39752) 
just comment.
 2024-08-22 22:16:21 +08:00