database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,891 Commits 1 Branch 0 Tags
eeacca9e8eaafb4431e11896cbf0be2f0cd60bfd
Commit Graph

9037 Commits

Author SHA1 Message Date
Socrates
eeacca9e8e [cherry-pick](branch-2.1) fix hudi count bug (#39785) 
bp #39656
 2024-08-22 22:35:44 +08:00
morrySnow
29898d0dbc [fix](downgrade) partition key should write upper case null type to image (#39776) 2024-08-22 18:42:08 +08:00
Xinyi Zou
40500f2b10 [branch-2.1](thirdparty) upgrade arrow to 17.0.0 (#39773) 
pick #38572
 2024-08-22 18:35:32 +08:00
zhangdong
f553645a71 [fix](mtmv) transfer col in mysql varchar to text when create MTMV (#37668) (#39727) 
pick from master #37668
 2024-08-22 15:20:59 +08:00
xzj7019
10f3e88f7a [fix](nereids) fix distribution expr list (#39435) 
pick from #39148
 2024-08-22 15:19:51 +08:00
wangbo
1c566253a8 [Pick][Improment]Query queued by be memory (#37559) (#39733) 
pick #37559
 2024-08-22 15:14:47 +08:00
toms
fd13962015 [chore](nereids) Added compatibility with mysql alias conflict (#38104) (#38440) 
throw table name/alias conflict exception to keep same behavior with mysql

for example:
```sql
select * from test.a b, test.b
```

error:
```
Not unique table/alias: 'b'
```
 2024-08-22 14:37:49 +08:00
toms
50f440e653 [chore](nereids) Added compatibility with mysql alias filter (#39738) 
qt_filter_select4 """
       select * from filter_alias_test.test b where filter_alias_test.b.id = 1;
    """

    qt_filter_select5 """
         select * from internal.filter_alias_test.test b where internal.filter_alias_test.b.id = 1;
    """
 2024-08-22 14:36:14 +08:00
morrySnow
83acdc14f4 [fix](Nereids) should push project through limit after eliminate union node (#39640) (#39755) 
pick from master #39640

otherwise:

push limit through union could generate plan:

limit
+-- union
    |-- limit
    |   +-- empty relation
    +-- limit
        +-- project

and then eliminate union will generate plan:

+-- limit
    +-  project
        +-- limit
            +-- project

it could not be processed by tranlator correctly
 2024-08-22 13:57:39 +08:00
LiBinfeng
ca9e50e49d [fix](Nereids) fix insert into table with null literal default value (#39122) (#39669) 
cherry-pick: #39122

Problem:
when use insert with default value null, it can not be insert
successfully
Solved:
when column is allow to be null, it can be null in create table with
null default value

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-22 10:37:50 +08:00
xzj7019
8f580b523f [opt](nereids) support partitionTopn for multi window exprs (#39687) 
## Proposed changes

pick from https://github.com/apache/doris/pull/38393

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-22 10:34:36 +08:00
seawinde
021982fc71 [fix](mtmv) Fix some pr to 21, prs are (#39041)(#38958)(#39541) (#39678) 
## Proposed changes

pr: https://github.com/apache/doris/pull/39041
commitId: 22562985

pr: https://github.com/apache/doris/pull/38958
commitId: c365cb64

pr: https://github.com/apache/doris/pull/39541
commitId: 89bb669c
 2024-08-22 10:27:55 +08:00
zhengyu
d36ba97eae [fix](statistics) correct update rows when doing multi-table load (#39548) 
rows of only one table is updated correctly, need to merge all table
commit infos.
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
 2024-08-22 09:57:45 +08:00
wangbo
ed9794a0fe [Pick][Improment]publish workload to BE by tag (#38486) (#39730) 
A workload group's tag property may be three cases as below: 1 empty
string, null or '', it could be published to all BE. 2 a value match
some BE' location, then the workload group could only be published to
the BE with same tag.
3 not an empty string, but some invalid string which can not math any
BE's location, then it could not be published any BE.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-22 00:48:16 +08:00
morrySnow
92671cbb73 [opt](Nereids) do not fallback if nereids failed because timeout (#39499) (#39718) 
pick from master #39499

since legacy planner will cost more time to plan, fallback will be worse
than throw exception directly
 2024-08-22 00:45:23 +08:00
wangbo
a55e109e97 [pick][Improment]Add schema table workload_group_privileges (#38436) (#39708) 
pick #38436
 2024-08-22 00:44:43 +08:00
Pxl
1e47d11560 [Improvement](runtime-filter) send RUNTIME_BLOOM_FILTER_MAX_SIZE to backends (#39686) 
…ackends (#38972)

## Proposed changes
pick from #38972
 2024-08-22 00:37:25 +08:00
camby
b5bdc325af [fix](profile) task type not the same in observer and master (#39245) (#39698) 
pick #39245 to branch-2.1
 2024-08-22 00:31:52 +08:00
shee
76b7a5291a [BUG] fix partition storage policy info lost (#38700) (#39677) 
## Proposed changes
  
cherry-pick from #38700 

Issue Number: close #xxx

<!--Describe your changes.-->

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: garenshi <garenshi@tencent.com>
 2024-08-22 00:23:19 +08:00
zhangdong
77270a0479 [enhance](mtmv)Improve the performance of obtaining partition/table v… (#39478) 
…ersions (#39301)
pick: https://github.com/apache/doris/pull/39301
 2024-08-22 00:07:52 +08:00
morrySnow
b878f7f1a6 [fix](planner) binary predicate result should compare with 0 (#39474) (#39717) 
pick from master #39474
 2024-08-21 20:47:14 +08:00
morrySnow
2890a98b18 [fix](mtmv) mtmv getPlanBySql should not reuse ctx's StatementContext (#39690) (#39719) 
pick from master #39690
 2024-08-21 19:20:40 +08:00
zy-kkk
80a27ca1bd [branch-2.1][improvement](jdbc catalog) Force all resources to be closed in the close method (#39666) 
pick (#39423)
 2024-08-21 18:35:35 +08:00
Jibing-Li
3a59ee1c5d [fix](auditlog)Record return row count in audit log for internal query. (#39616) (#39702) 
backport: https://github.com/apache/doris/pull/39616
 2024-08-21 17:37:01 +08:00
meiyi
1460878bdf [fix](cluster key) forbid cluster key and remove case (#39679) 
branch-2.1 does not support mow cluster key
 2024-08-21 14:31:54 +08:00
starocean999
ba3b56d269 [fix](nereids)prevent null pointer exception if datetime value overflows (#39675) 
pick from master https://github.com/apache/doris/pull/39482
 2024-08-21 14:17:34 +08:00
Mingyu Chen
0bfcee1251 [opt](file-cache) support system table file_cache_statistics (#39552) 
1. Add new system table: `file_cache_statistics`

	This table is used for viewing metrics related to file cache on BE side

	```
	mysql> select * from information_schema.file_cache_statistics limit 10;

+-------+---------------+----------------------------+--------------------------------+--------------------+
| BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE |

+-------+---------------+----------------------------+--------------------------------+--------------------+
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_elements | 102400 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_size | 21474836480 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio |
0.8539634687001242 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_max_elements | 102400 |

+-------+---------------+----------------------------+--------------------------------+--------------------+
	```

	It will show metrics of file caches on each BE.

2. Add new metrics `hits_ratio_1h` and `hits_ratio_5m` for file cache

This 2 metrics will show the hit ratio of file cache in recent 1 hour or
5 minutes.
So that we can know recent hit ratio instead of global historical hit
ratio.
 2024-08-21 10:03:39 +08:00
wuwenchi
bf26f49505 [bugfix](external)add check of engine and catalog types for 2.1 #39343 (#39643) 
bp #39343
 2024-08-21 09:50:17 +08:00
xzj7019
8a562aeb77 [opt](nereids) recover adoptive bucket shuffle (#39598) 
## Proposed changes

pick from https://github.com/apache/doris/pull/36784

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-21 09:26:53 +08:00
wuwenchi
6df6f1dc97 [improvement](iceberg)]support doris's char/varchar to iceberg's string for 2.1 #38807 (#39645) 
bp: #38807
 2024-08-21 09:19:10 +08:00
wuwenchi
28ce116b17 [improvement](iceberg)add some description for show create for 2.1 #39179 (#39644) 
## Proposed changes

bp: #39179

1. add `location` and `properties` for `show create table`.
2. add `location` for `show create database`.
 2024-08-21 09:18:38 +08:00
yujun
2fe6d580be [improvement](diagnose) add tablet in recycle bin hint #39547 (#39622) 
cherry pick from #39547
 2024-08-21 09:16:01 +08:00
yujun
57262a3d5c [fix](partition rebalancer) fix migrate tablets between backends back and forth #39333 (#39606) 
cherry pick from #39333
 2024-08-21 09:15:31 +08:00
Socrates
bb687bd69c [cherry-pick](branch-2.1) add function regexp_extract_or_null (#39561) 
# Proposed changes

pick https://github.com/apache/doris/pull/38296
 2024-08-21 09:14:58 +08:00
Jibing-Li
7c3c5c67fc [log](statistics)Add result row count log for statistics internal query. (#39556) (#39609) 
backport: https://github.com/apache/doris/pull/39556
 2024-08-20 23:18:48 +08:00
Mingyu Chen
a5daa3edc8 [opt](variables) enlarge the default value of max_allowed_packet (#38697) (#39626) 
bp #38697
 2024-08-20 22:02:01 +08:00
Mingyu Chen
dfd21bd2a0 [fix](fe-log) add position info in async mode #39419 (#39571) 
pick part of #39419
 2024-08-20 22:01:34 +08:00
Mingyu Chen
a4deefea5d [fix](catalog) gen partition id by name (#39325) (#39625) 
bp #39325
 2024-08-20 22:00:19 +08:00
Mingyu Chen
a3fd13fee6 [fix](catalog) set timeout for split fetch (#39346) (#39624) 
bp #39346
 2024-08-20 21:59:55 +08:00
Mingyu Chen
0e21dba817 [opt](catalog) modify some meta cache logic (#38506) (#39628) 
#38506
 2024-08-20 21:57:55 +08:00
yujun
607887673e [improvement](report) report handler discard old report tasks #39469 (#39605) 
cherry pick from #39469
 2024-08-20 17:40:49 +08:00
Gabriel
e302882e52 [branch-2.1](pick) Pick 2 PRs to branch-2.1 (#39604) 
## Proposed changes

pick #39480 #39589

<!--Describe your changes.-->
 2024-08-20 17:10:30 +08:00
Gavin Chou
621d394a5e [enhance](Backup) Do connectivity check when creating repository (#38350) (#39538) 
Previously when creating repository, FE would not do connectivity check.
It might result in confusing error when using backup restore.

pick #38350

Co-authored-by: AlexYue <yj976240184@gmail.com>
 2024-08-19 22:16:02 +08:00
hui lai
9647885b95 [fix](routine load) should update progress before handle transaction state transform (#39311) (#39526) 
pick (#39311)

Update progress maybe throw exception, causing offset has been persisted
on edit log or meta service, but the memory data has not been updated.
It will cause repeated consumption.
 2024-08-19 21:23:59 +08:00
hui lai
3d8b04a782 [fix](stream load) do not throw exception but skip record when can not find database (#39360) (#39527) 
pick (#39360)

When fetch stream load record from BE node, if can not find database,
StreamLoadRecordMgr will throw exception and the remaining records will
not be recorded in memory.

For example: Ten stream load records were pulled, and the database
associated with the stream load of the first record was deleted by the
user. Therefore, the pull will end, resulting in the remaining nine
records not being consumed recorded in memory.

This pr do not throw exception but skip record when can not find
database to solve this problem.
 2024-08-19 21:23:26 +08:00
hui lai
a1aa9b8ab9 [fix](routine load) add read lock to fix some concurrent bugs (#39242) (#39525) 
pick #39242
 2024-08-19 21:18:27 +08:00
zhiqiang
830f250a80 [opt](query cancel) cancel query if it has pipeline task leakage #39223 (#39537) 
pick #39223 with some modifications. Optimization will only be applied
to pipeline x.
 2024-08-19 14:33:59 +08:00
Siyang Tang
c0cbb2362c [enhancement](schema-change) Record detailed fail reason for schema change tasks (#39351) (#39501) 
## Proposed changes

Expose the error msg from BE as the real fail reason recorded for schema
change tasks. To avoid too much memory usage, we just pick one among all
to record.
 2024-08-18 13:51:06 +08:00
wuwenchi
e01d051acf [improvement](external catalog)Optimize the process of refreshing catalog for 2.1 (#39205) (#39186) 
## Proposed changes

bp: #39205

When the catalog attributes have not changed, refreshing the catalog
only requires processing the cache, without rebuilding the entire
catalog.
 2024-08-17 17:02:06 +08:00
walter
fd4d1f4e4f [chore](table) Add batch method to get visible version of the olap table (#38949) (#39495) 
Cherry-pick #38949
 2024-08-17 16:55:06 +08:00