database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,941 Commits 1 Branch 0 Tags
f2a37d58fb86ef09165ac8d34535926c47517cb3
Commit Graph

9059 Commits

Author SHA1 Message Date
LiBinfeng
9997911ec9 [feat](Nereids) support nereids hint position detaction (#39113) (#39417) 
cherry-pick: #39113
When use hint in wrong position or use unsupport hint, use channel(2) to
filter it out

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-24 23:59:54 +08:00
Gavin Chou
8e140727ae [fix](udf) Fix exception thrown when replayDropFunction (#39868) 
This PR completes #25965 to fix uncaught runtime exception when calling
`org.apache.doris.catalog.GlobalFunctionMgr#replayDropFunction`


![img_v3_02e0_3889b1de-583e-48ba-8e7f-48e1d3ae799g](https://github.com/user-attachments/assets/b0dbfefb-16f3-4dbb-8a00-f4773978ba01)
 2024-08-24 23:56:48 +08:00
TengJianPing
2dea859bdb [debug](rpc) debug rpc time consumption problem (#39852) 
## Proposed changes

Issue Number: close #xxx

Add detail RPC time info for each channel, sorted by max rpc time of
channels:
```
                     DATA_STREAM_SINK_OPERATOR  (id=1,dst_id=1):
                          -  Partitioner:  Crc32HashPartitioner(64)
                          -  BlocksProduced:  74
                          -  BrpcSendTime:  2.689us
                          -  BrpcSendTime.Wait:  0ns
                          -  BytesSent:  89.35  KB
                          -  CloseTime:  680.152us
                          -  CompressTime:  0ns
                          -  ExecTime:  160.663ms
                          -  InitTime:  263.608us
                          -  InputRows:  32.512K  (32512)
                          -  LocalBytesSent:  0.00  
                          -  LocalSendTime:  0ns
                          -  LocalSentRows:  0
                          -  MemoryUsage:  
                              -  PeakMemoryUsage:  80.00  KB
                          -  MergeBlockTime:  0ns
                          -  OpenTime:  4.113ms
                          -  OverallThroughput:  0.0  /sec
                          -  PendingFinishDependency:  41.179ms
                          -  RowsProduced:  32.512K  (32512)
                          -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m
```
 2024-08-24 19:59:39 +08:00
Mingyu Chen
14a2a66106 [fix](paimon) fix not able to read paimon data from hdfs with HA (#39806) (#39876) 
bp #39806
 2024-08-24 17:51:15 +08:00
zclllhhjj
460605ae3c [branch-2.1] pick some prs (#39860) 
## Proposed changes

Issue Number: close #xxx

https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
 2024-08-24 17:26:42 +08:00
Mingyu Chen
9640e2de44 [fix](catalog) refactor location path and support default fs #39116 (#39203) 2024-08-24 16:05:13 +08:00
Mingyu Chen
5ed56770d4 [bugfix](external) Prevent multiple fs from being generated (#39663) (#39870) 
bp #39663

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
 2024-08-24 14:17:26 +08:00
Mingyu Chen
de2e8f0ae6 [fix](ctas) fix NPE when ctas with old planner and varchar issue (#39744) (#39871) 
bp #39744
 2024-08-24 09:24:47 +08:00
Mingyu Chen
b9da934b16 [fix](hive) report error with escape char and null format (#39700) (#39869) 
bp #39700

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-08-24 09:23:03 +08:00
zy-kkk
4cb7122a89 [branch-2.1][fix](jdbc catalog) Fix Memory Leak by Enabling Weak References in HikariCP (#39833) 
pick(#39582)
 2024-08-24 00:45:12 +08:00
Siyang Tang
00e4f343b0 [enhancement](delete) Add a hint msg for forbidden delete when MV or rollup exists (#39505) (#39857) 
## Proposed changes

When MV or Rollup exists, delete is forbidden on the base table
currently. Add a hint msg to indicate it.
 2024-08-23 23:42:26 +08:00
morrySnow
9d597bde68 [fix](Nereids) producer to consumer should be multimap in cte (#39850) (#39867) 2024-08-23 23:25:11 +08:00
Mingyu Chen
508c7a7040 [fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. (#39239) (#39865) 
bp #39239

Co-authored-by: daidai <2017501503@qq.com>
 2024-08-23 23:21:02 +08:00
zhangdong
19c51cc217 [fix](mtmv) use name instead of id in meta of MTMV (#39355) (#39748) 
pick from master #39355
 2024-08-23 20:40:54 +08:00
wangbo
6ceb574aa0 [branch-2.1]Pick IO limit/workload group usage table (#39839) 2024-08-23 18:51:47 +08:00
Xinyi Zou
e716658fba [branch-2.1](arrow-flight-sql) Fix exceed user property max connection cause Reach limit of connections (#39836) 
pick #39127
pick #39802
 2024-08-23 17:27:34 +08:00
xzj7019
424ad2384a [opt](nereids) refine left semi/anti cost under short-cut opt (#39636) 
## Proposed changes

pick from https://github.com/apache/doris/pull/37951

---------

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-23 17:26:56 +08:00
Xujian Duan
8f15efdbb8 [cherry-pick](branch-2.1) fix delete random distributed tbl (#39830) 
## Proposed changes

cherry-pick #37985

<!--Describe your changes.-->
 2024-08-23 17:17:05 +08:00
wangbo
7a7292ad5a [branch-2.1][Refactor]use async to get be resource (#38389) (#39826) 
pick #38389
 2024-08-23 17:16:19 +08:00
slothever
67a8099991 [fix](multi-catalog)fix max compute array and map type read offset (#39822) 
bp #39680
 2024-08-23 16:53:52 +08:00
yujun
0934fbee7e [improvement](query) prefer to chose tablet on alive disk #39467 (#39654) 
cherry pick from #39467
 2024-08-23 12:23:12 +08:00
htyoung
6c10c47f79 [fix](fe) LIST partition table support modify default bucket num (#39688) 
## Proposed changes
bp #39696

Issue Number: close #39684

```sql
CREATE TABLE `test1` (
    `id1` VARCHAR(255) NULL COMMENT 'id1',
    `id2` VARCHAR(255) NULL COMMENT 'id2',
    `event_time` VARCHAR(255) NULL COMMENT '事件时间',
    `event_date` VARCHAR(255) NULL COMMENT '事件日期',
    `event_ts` VARCHAR(256) NULL COMMENT '事件发生时间戳(毫秒)',
    `dt` VARCHAR(255) NOT NULL COMMENT '日期分区',
    `hr` VARCHAR(255) NOT NULL COMMENT '小时分区'
  ) ENGINE = OLAP DUPLICATE KEY(`id1`) COMMENT 'xxx' PARTITION BY LIST(`dt`, `hr`) (
    PARTITION p2024082021 VALUES IN (("2024-08-20", "21"))
  ) DISTRIBUTED BY HASH(`dt`, `hr`) BUCKETS 2 PROPERTIES (
    "replication_allocation" = "tag.location.default: 1",
    "min_load_replica_num" = "-1",
    "is_being_synced" = "false",
    "storage_medium" = "hdd",
    "storage_format" = "V2",
    "inverted_index_storage_format" = "V1",
    "light_schema_change" = "true",
    "disable_auto_compaction" = "false",
    "enable_single_replica_compaction" = "false",
    "group_commit_interval_ms" = "10000",
    "group_commit_data_bytes" = "134217728"
  );
```

1. 修改前表已有分区的bucket num为2

![image](https://github.com/user-attachments/assets/77efdd0c-f845-41a4-9a31-e454808ffe67)

2. 修改List分区表bucket num（从2 -> 4）

![image](https://github.com/user-attachments/assets/53b19918-2879-4cb3-b2bd-84ba35a7fc59)

3. 修改List分区后，添加的新分区bucket num为4

![image](https://github.com/user-attachments/assets/1f41f73f-d70f-433e-a7b6-8346b7dfcc4e)

Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>
 2024-08-23 11:52:16 +08:00
Socrates
eeacca9e8e [cherry-pick](branch-2.1) fix hudi count bug (#39785) 
bp #39656
 2024-08-22 22:35:44 +08:00
morrySnow
29898d0dbc [fix](downgrade) partition key should write upper case null type to image (#39776) 2024-08-22 18:42:08 +08:00
Xinyi Zou
40500f2b10 [branch-2.1](thirdparty) upgrade arrow to 17.0.0 (#39773) 
pick #38572
 2024-08-22 18:35:32 +08:00
zhangdong
f553645a71 [fix](mtmv) transfer col in mysql varchar to text when create MTMV (#37668) (#39727) 
pick from master #37668
 2024-08-22 15:20:59 +08:00
xzj7019
10f3e88f7a [fix](nereids) fix distribution expr list (#39435) 
pick from #39148
 2024-08-22 15:19:51 +08:00
wangbo
1c566253a8 [Pick][Improment]Query queued by be memory (#37559) (#39733) 
pick #37559
 2024-08-22 15:14:47 +08:00
toms
fd13962015 [chore](nereids) Added compatibility with mysql alias conflict (#38104) (#38440) 
throw table name/alias conflict exception to keep same behavior with mysql

for example:
```sql
select * from test.a b, test.b
```

error:
```
Not unique table/alias: 'b'
```
 2024-08-22 14:37:49 +08:00
toms
50f440e653 [chore](nereids) Added compatibility with mysql alias filter (#39738) 
qt_filter_select4 """
       select * from filter_alias_test.test b where filter_alias_test.b.id = 1;
    """

    qt_filter_select5 """
         select * from internal.filter_alias_test.test b where internal.filter_alias_test.b.id = 1;
    """
 2024-08-22 14:36:14 +08:00
morrySnow
83acdc14f4 [fix](Nereids) should push project through limit after eliminate union node (#39640) (#39755) 
pick from master #39640

otherwise:

push limit through union could generate plan:

limit
+-- union
    |-- limit
    |   +-- empty relation
    +-- limit
        +-- project

and then eliminate union will generate plan:

+-- limit
    +-  project
        +-- limit
            +-- project

it could not be processed by tranlator correctly
 2024-08-22 13:57:39 +08:00
LiBinfeng
ca9e50e49d [fix](Nereids) fix insert into table with null literal default value (#39122) (#39669) 
cherry-pick: #39122

Problem:
when use insert with default value null, it can not be insert
successfully
Solved:
when column is allow to be null, it can be null in create table with
null default value

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-22 10:37:50 +08:00
xzj7019
8f580b523f [opt](nereids) support partitionTopn for multi window exprs (#39687) 
## Proposed changes

pick from https://github.com/apache/doris/pull/38393

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-22 10:34:36 +08:00
seawinde
021982fc71 [fix](mtmv) Fix some pr to 21, prs are (#39041)(#38958)(#39541) (#39678) 
## Proposed changes

pr: https://github.com/apache/doris/pull/39041
commitId: 22562985

pr: https://github.com/apache/doris/pull/38958
commitId: c365cb64

pr: https://github.com/apache/doris/pull/39541
commitId: 89bb669c
 2024-08-22 10:27:55 +08:00
zhengyu
d36ba97eae [fix](statistics) correct update rows when doing multi-table load (#39548) 
rows of only one table is updated correctly, need to merge all table
commit infos.
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
 2024-08-22 09:57:45 +08:00
wangbo
ed9794a0fe [Pick][Improment]publish workload to BE by tag (#38486) (#39730) 
A workload group's tag property may be three cases as below: 1 empty
string, null or '', it could be published to all BE. 2 a value match
some BE' location, then the workload group could only be published to
the BE with same tag.
3 not an empty string, but some invalid string which can not math any
BE's location, then it could not be published any BE.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-22 00:48:16 +08:00
morrySnow
92671cbb73 [opt](Nereids) do not fallback if nereids failed because timeout (#39499) (#39718) 
pick from master #39499

since legacy planner will cost more time to plan, fallback will be worse
than throw exception directly
 2024-08-22 00:45:23 +08:00
wangbo
a55e109e97 [pick][Improment]Add schema table workload_group_privileges (#38436) (#39708) 
pick #38436
 2024-08-22 00:44:43 +08:00
Pxl
1e47d11560 [Improvement](runtime-filter) send RUNTIME_BLOOM_FILTER_MAX_SIZE to backends (#39686) 
…ackends (#38972)

## Proposed changes
pick from #38972
 2024-08-22 00:37:25 +08:00
camby
b5bdc325af [fix](profile) task type not the same in observer and master (#39245) (#39698) 
pick #39245 to branch-2.1
 2024-08-22 00:31:52 +08:00
shee
76b7a5291a [BUG] fix partition storage policy info lost (#38700) (#39677) 
## Proposed changes
  
cherry-pick from #38700 

Issue Number: close #xxx

<!--Describe your changes.-->

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: garenshi <garenshi@tencent.com>
 2024-08-22 00:23:19 +08:00
zhangdong
77270a0479 [enhance](mtmv)Improve the performance of obtaining partition/table v… (#39478) 
…ersions (#39301)
pick: https://github.com/apache/doris/pull/39301
 2024-08-22 00:07:52 +08:00
morrySnow
b878f7f1a6 [fix](planner) binary predicate result should compare with 0 (#39474) (#39717) 
pick from master #39474
 2024-08-21 20:47:14 +08:00
morrySnow
2890a98b18 [fix](mtmv) mtmv getPlanBySql should not reuse ctx's StatementContext (#39690) (#39719) 
pick from master #39690
 2024-08-21 19:20:40 +08:00
zy-kkk
80a27ca1bd [branch-2.1][improvement](jdbc catalog) Force all resources to be closed in the close method (#39666) 
pick (#39423)
 2024-08-21 18:35:35 +08:00
Jibing-Li
3a59ee1c5d [fix](auditlog)Record return row count in audit log for internal query. (#39616) (#39702) 
backport: https://github.com/apache/doris/pull/39616
 2024-08-21 17:37:01 +08:00
meiyi
1460878bdf [fix](cluster key) forbid cluster key and remove case (#39679) 
branch-2.1 does not support mow cluster key
 2024-08-21 14:31:54 +08:00
starocean999
ba3b56d269 [fix](nereids)prevent null pointer exception if datetime value overflows (#39675) 
pick from master https://github.com/apache/doris/pull/39482
 2024-08-21 14:17:34 +08:00
Mingyu Chen
0bfcee1251 [opt](file-cache) support system table file_cache_statistics (#39552) 
1. Add new system table: `file_cache_statistics`

	This table is used for viewing metrics related to file cache on BE side

	```
	mysql> select * from information_schema.file_cache_statistics limit 10;

+-------+---------------+----------------------------+--------------------------------+--------------------+
| BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE |

+-------+---------------+----------------------------+--------------------------------+--------------------+
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_elements | 102400 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_size | 21474836480 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio |
0.8539634687001242 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_max_elements | 102400 |

+-------+---------------+----------------------------+--------------------------------+--------------------+
	```

	It will show metrics of file caches on each BE.

2. Add new metrics `hits_ratio_1h` and `hits_ratio_5m` for file cache

This 2 metrics will show the hit ratio of file cache in recent 1 hour or
5 minutes.
So that we can know recent hit ratio instead of global historical hit
ratio.
 2024-08-21 10:03:39 +08:00
wuwenchi
bf26f49505 [bugfix](external)add check of engine and catalog types for 2.1 #39343 (#39643) 
bp #39343
 2024-08-21 09:50:17 +08:00