database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,891 Commits 1 Branch 0 Tags
eeacca9e8eaafb4431e11896cbf0be2f0cd60bfd
Commit Graph

19891 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Socrates
eeacca9e8e [cherry-pick](branch-2.1) fix hudi count bug (#39785) 
bp #39656
 2024-08-22 22:35:44 +08:00
zhiqiang
06a0b35704 [chore] Comment for tv_nsec (#39752) 
just comment.
 2024-08-22 22:16:21 +08:00
Mryange
04e993c1de [refine](pipeline) refine some VDataStreamRecvr code (#35063) (#37802) 
## Proposed changes
https://github.com/apache/doris/pull/35063
https://github.com/apache/doris/pull/35428
 2024-08-22 19:55:17 +08:00
morrySnow
29898d0dbc [fix](downgrade) partition key should write upper case null type to image (#39776) 2024-08-22 18:42:08 +08:00
Xinyi Zou
40500f2b10 [branch-2.1](thirdparty) upgrade arrow to 17.0.0 (#39773) 
pick #38572
 2024-08-22 18:35:32 +08:00
Xinyi Zou
13b882a4cc [branch-2.1](memory) Add memory metrics to bvar (#39763) 
pick #38391
 2024-08-22 17:34:30 +08:00
Xinyi Zou
8ce8887b75 [branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth (#39760) 
pick #38168
overwrites changes in #37221 on workload_group_manager.cpp. If need to
pick 37221, ignore it.
 2024-08-22 17:33:11 +08:00
TengJianPing
ba7baa7e6b [fix](window_funnel) fix upgrading problem caused by behaviour change of window_funnel (#39766) 
## Proposed changes

Issue Number: close #xxx

For the latest published 2.1 version `2.1.5`:
```
max_be_exec_version=4;
AGG_FUNCTION_NEW=2;
```
and `branch-2.1`:
```
max_be_exec_version=5;
AGG_FUNCTION_NEW=2;
```
It will cause problem when upgrading.
This PR fix the problem, set `AGG_FUNCTION_NEW` to `5`.
 2024-08-22 17:26:51 +08:00
zhangdong
f553645a71 [fix](mtmv) transfer col in mysql varchar to text when create MTMV (#37668) (#39727) 
pick from master #37668
 2024-08-22 15:20:59 +08:00
xzj7019
10f3e88f7a [fix](nereids) fix distribution expr list (#39435) 
pick from #39148
 2024-08-22 15:19:51 +08:00
wangbo
1c566253a8 [Pick][Improment]Query queued by be memory (#37559) (#39733) 
pick #37559
 2024-08-22 15:14:47 +08:00
toms
fd13962015 [chore](nereids) Added compatibility with mysql alias conflict (#38104) (#38440) 
throw table name/alias conflict exception to keep same behavior with mysql

for example:
```sql
select * from test.a b, test.b
```

error:
```
Not unique table/alias: 'b'
```
 2024-08-22 14:37:49 +08:00
toms
50f440e653 [chore](nereids) Added compatibility with mysql alias filter (#39738) 
qt_filter_select4 """
       select * from filter_alias_test.test b where filter_alias_test.b.id = 1;
    """

    qt_filter_select5 """
         select * from internal.filter_alias_test.test b where internal.filter_alias_test.b.id = 1;
    """
 2024-08-22 14:36:14 +08:00
morrySnow
83acdc14f4 [fix](Nereids) should push project through limit after eliminate union node (#39640) (#39755) 
pick from master #39640

otherwise:

push limit through union could generate plan:

limit
+-- union
    |-- limit
    |   +-- empty relation
    +-- limit
        +-- project

and then eliminate union will generate plan:

+-- limit
    +-  project
        +-- limit
            +-- project

it could not be processed by tranlator correctly
 2024-08-22 13:57:39 +08:00
LiBinfeng
ca9e50e49d [fix](Nereids) fix insert into table with null literal default value (#39122) (#39669) 
cherry-pick: #39122

Problem:
when use insert with default value null, it can not be insert
successfully
Solved:
when column is allow to be null, it can be null in create table with
null default value

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-22 10:37:50 +08:00
xzj7019
8f580b523f [opt](nereids) support partitionTopn for multi window exprs (#39687) 
## Proposed changes

pick from https://github.com/apache/doris/pull/38393

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-22 10:34:36 +08:00
seawinde
021982fc71 [fix](mtmv) Fix some pr to 21, prs are (#39041)(#38958)(#39541) (#39678) 
## Proposed changes

pr: https://github.com/apache/doris/pull/39041
commitId: 22562985

pr: https://github.com/apache/doris/pull/38958
commitId: c365cb64

pr: https://github.com/apache/doris/pull/39541
commitId: 89bb669c
 2024-08-22 10:27:55 +08:00
zhengyu
d36ba97eae [fix](statistics) correct update rows when doing multi-table load (#39548) 
rows of only one table is updated correctly, need to merge all table
commit infos.
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
 2024-08-22 09:57:45 +08:00
wangbo
ed9794a0fe [Pick][Improment]publish workload to BE by tag (#38486) (#39730) 
A workload group's tag property may be three cases as below: 1 empty
string, null or '', it could be published to all BE. 2 a value match
some BE' location, then the workload group could only be published to
the BE with same tag.
3 not an empty string, but some invalid string which can not math any
BE's location, then it could not be published any BE.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-22 00:48:16 +08:00
morrySnow
92671cbb73 [opt](Nereids) do not fallback if nereids failed because timeout (#39499) (#39718) 
pick from master #39499

since legacy planner will cost more time to plan, fallback will be worse
than throw exception directly
 2024-08-22 00:45:23 +08:00
wangbo
a55e109e97 [pick][Improment]Add schema table workload_group_privileges (#38436) (#39708) 
pick #38436
 2024-08-22 00:44:43 +08:00
zhannngchen
0e694f19db [fix](merge-on-write) segcompaction should process delete bitmap if necessary (#38369) (#39707) 
## Proposed changes

Issue Number: close #xxx

cherry-pick #38369 and #38800
 2024-08-22 00:42:56 +08:00
Vallish Pai
935d0eb110 [cherry-pick](branch-2.1) [Bug](http-api) fix core dump on API check_rpc_channel coz exec_env not initialized #39519 (#39692) 
…rpc_channel coz exec_env not initialized #39519

## Proposed changes

Issue Number: close #xxx
backport #39519  #39520
 2024-08-22 00:42:12 +08:00
Tiewei Fang
a613627a00 [fix](test) fix p2 regression test of export (#39683) 
bp: #39088
 2024-08-22 00:41:43 +08:00
zhiqiang
56cc9cc304 [fix](cancel)) Fix pipeline task leak cancel (#39697) 
pick #39737
 2024-08-22 00:40:22 +08:00
Pxl
1e47d11560 [Improvement](runtime-filter) send RUNTIME_BLOOM_FILTER_MAX_SIZE to backends (#39686) 
…ackends (#38972)

## Proposed changes
pick from #38972
 2024-08-22 00:37:25 +08:00
camby
b5bdc325af [fix](profile) task type not the same in observer and master (#39245) (#39698) 
pick #39245 to branch-2.1
 2024-08-22 00:31:52 +08:00
Pxl
5e91fc6a8f [Bug](runtime-filter) set inited to true on BloomFilterFuncBase::assi… (#39674) 
…gn (#39335)

## Proposed changes
pick from #39335
 2024-08-22 00:29:16 +08:00
Pxl
63d45f5d89 [Bug](predicate) fix wrong result of AcceptNullPredicate (#39497) (#39672) 
pick from #39497
 2024-08-22 00:24:57 +08:00
Mryange
e51dd68b93 [fix](local shuffle) Fix correctness for bucket hash shuffle exchange… (#39691) 
…r (#39568)

For query plan


![image](https://github.com/user-attachments/assets/334cc4c4-49ae-4330-83ff-03b9bae00e3c)

we will plan local exchangers  and get a new plan


![image](https://github.com/user-attachments/assets/2b8ece64-3aa0-423c-9db0-fd02024957db)

and the hash join operator will get probe and build data which are
different distributed (one is HASH shuffle and another is Bucket hash
shuffle). This PR fix it.
<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
 2024-08-22 00:23:39 +08:00
shee
76b7a5291a [BUG] fix partition storage policy info lost (#38700) (#39677) 
## Proposed changes
  
cherry-pick from #38700 

Issue Number: close #xxx

<!--Describe your changes.-->

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: garenshi <garenshi@tencent.com>
 2024-08-22 00:23:19 +08:00
zhangdong
77270a0479 [enhance](mtmv)Improve the performance of obtaining partition/table v… (#39478) 
…ersions (#39301)
pick: https://github.com/apache/doris/pull/39301
 2024-08-22 00:07:52 +08:00
morrySnow
b878f7f1a6 [fix](planner) binary predicate result should compare with 0 (#39474) (#39717) 
pick from master #39474
 2024-08-21 20:47:14 +08:00
morrySnow
2890a98b18 [fix](mtmv) mtmv getPlanBySql should not reuse ctx's StatementContext (#39690) (#39719) 
pick from master #39690
 2024-08-21 19:20:40 +08:00
zy-kkk
80a27ca1bd [branch-2.1][improvement](jdbc catalog) Force all resources to be closed in the close method (#39666) 
pick (#39423)
 2024-08-21 18:35:35 +08:00
slothever
ebbebdf590 [regression](kerberos)add hive with kerberos write back case (#39682) 
bp #38647
 2024-08-21 18:29:42 +08:00
Jibing-Li
3a59ee1c5d [fix](auditlog)Record return row count in audit log for internal query. (#39616) (#39702) 
backport: https://github.com/apache/doris/pull/39616
 2024-08-21 17:37:01 +08:00
Yongqiang YANG
610f69432a [improvement](segmentcache) limit segment cache by fd limit or memory… (#39689) 
… (#39658)

remove a useless config.
 2024-08-21 15:19:52 +08:00
zhangstar333
1e30d4ebaf Revert "[Bug](compatibility) fix window funnel function coredump when upgrade" (#39681) 
Reverts apache/doris#39646
 2024-08-21 14:47:27 +08:00
meiyi
1460878bdf [fix](cluster key) forbid cluster key and remove case (#39679) 
branch-2.1 does not support mow cluster key
 2024-08-21 14:31:54 +08:00
starocean999
ba3b56d269 [fix](nereids)prevent null pointer exception if datetime value overflows (#39675) 
pick from master https://github.com/apache/doris/pull/39482
 2024-08-21 14:17:34 +08:00
Mingyu Chen
0bfcee1251 [opt](file-cache) support system table file_cache_statistics (#39552) 
1. Add new system table: `file_cache_statistics`

	This table is used for viewing metrics related to file cache on BE side

	```
	mysql> select * from information_schema.file_cache_statistics limit 10;

+-------+---------------+----------------------------+--------------------------------+--------------------+
| BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE |

+-------+---------------+----------------------------+--------------------------------+--------------------+
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_elements | 102400 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_size | 21474836480 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio |
0.8539634687001242 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_max_elements | 102400 |

+-------+---------------+----------------------------+--------------------------------+--------------------+
	```

	It will show metrics of file caches on each BE.

2. Add new metrics `hits_ratio_1h` and `hits_ratio_5m` for file cache

This 2 metrics will show the hit ratio of file cache in recent 1 hour or
5 minutes.
So that we can know recent hit ratio instead of global historical hit
ratio.
 2024-08-21 10:03:39 +08:00
wuwenchi
bf26f49505 [bugfix](external)add check of engine and catalog types for 2.1 #39343 (#39643) 
bp #39343
 2024-08-21 09:50:17 +08:00
xzj7019
8a562aeb77 [opt](nereids) recover adoptive bucket shuffle (#39598) 
## Proposed changes

pick from https://github.com/apache/doris/pull/36784

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-21 09:26:53 +08:00
wuwenchi
6df6f1dc97 [improvement](iceberg)]support doris's char/varchar to iceberg's string for 2.1 #38807 (#39645) 
bp: #38807
 2024-08-21 09:19:10 +08:00
wuwenchi
28ce116b17 [improvement](iceberg)add some description for show create for 2.1 #39179 (#39644) 
## Proposed changes

bp: #39179

1. add `location` and `properties` for `show create table`.
2. add `location` for `show create database`.
 2024-08-21 09:18:38 +08:00
daidai
27ba2542e2 [case](iceberg)append iceberg schema change case. (#38766) (#39630) 
bp #38766
 2024-08-21 09:17:12 +08:00
yujun
2fe6d580be [improvement](diagnose) add tablet in recycle bin hint #39547 (#39622) 
cherry pick from #39547
 2024-08-21 09:16:01 +08:00
yujun
57262a3d5c [fix](partition rebalancer) fix migrate tablets between backends back and forth #39333 (#39606) 
cherry pick from #39333
 2024-08-21 09:15:31 +08:00
Socrates
bb687bd69c [cherry-pick](branch-2.1) add function regexp_extract_or_null (#39561) 
# Proposed changes

pick https://github.com/apache/doris/pull/38296
 2024-08-21 09:14:58 +08:00