database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,867 Commits 1 Branch 0 Tags
56cc9cc304847a574116be6bc0c166fbe23f2f80
Commit Graph

19867 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zhiqiang
56cc9cc304 [fix](cancel)) Fix pipeline task leak cancel (#39697) 
pick #39737
 2024-08-22 00:40:22 +08:00
Pxl
1e47d11560 [Improvement](runtime-filter) send RUNTIME_BLOOM_FILTER_MAX_SIZE to backends (#39686) 
…ackends (#38972)

## Proposed changes
pick from #38972
 2024-08-22 00:37:25 +08:00
camby
b5bdc325af [fix](profile) task type not the same in observer and master (#39245) (#39698) 
pick #39245 to branch-2.1
 2024-08-22 00:31:52 +08:00
Pxl
5e91fc6a8f [Bug](runtime-filter) set inited to true on BloomFilterFuncBase::assi… (#39674) 
…gn (#39335)

## Proposed changes
pick from #39335
 2024-08-22 00:29:16 +08:00
Pxl
63d45f5d89 [Bug](predicate) fix wrong result of AcceptNullPredicate (#39497) (#39672) 
pick from #39497
 2024-08-22 00:24:57 +08:00
Mryange
e51dd68b93 [fix](local shuffle) Fix correctness for bucket hash shuffle exchange… (#39691) 
…r (#39568)

For query plan


![image](https://github.com/user-attachments/assets/334cc4c4-49ae-4330-83ff-03b9bae00e3c)

we will plan local exchangers  and get a new plan


![image](https://github.com/user-attachments/assets/2b8ece64-3aa0-423c-9db0-fd02024957db)

and the hash join operator will get probe and build data which are
different distributed (one is HASH shuffle and another is Bucket hash
shuffle). This PR fix it.
<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
 2024-08-22 00:23:39 +08:00
shee
76b7a5291a [BUG] fix partition storage policy info lost (#38700) (#39677) 
## Proposed changes
  
cherry-pick from #38700 

Issue Number: close #xxx

<!--Describe your changes.-->

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: garenshi <garenshi@tencent.com>
 2024-08-22 00:23:19 +08:00
zhangdong
77270a0479 [enhance](mtmv)Improve the performance of obtaining partition/table v… (#39478) 
…ersions (#39301)
pick: https://github.com/apache/doris/pull/39301
 2024-08-22 00:07:52 +08:00
morrySnow
b878f7f1a6 [fix](planner) binary predicate result should compare with 0 (#39474) (#39717) 
pick from master #39474
 2024-08-21 20:47:14 +08:00
morrySnow
2890a98b18 [fix](mtmv) mtmv getPlanBySql should not reuse ctx's StatementContext (#39690) (#39719) 
pick from master #39690
 2024-08-21 19:20:40 +08:00
zy-kkk
80a27ca1bd [branch-2.1][improvement](jdbc catalog) Force all resources to be closed in the close method (#39666) 
pick (#39423)
 2024-08-21 18:35:35 +08:00
slothever
ebbebdf590 [regression](kerberos)add hive with kerberos write back case (#39682) 
bp #38647
 2024-08-21 18:29:42 +08:00
Jibing-Li
3a59ee1c5d [fix](auditlog)Record return row count in audit log for internal query. (#39616) (#39702) 
backport: https://github.com/apache/doris/pull/39616
 2024-08-21 17:37:01 +08:00
Yongqiang YANG
610f69432a [improvement](segmentcache) limit segment cache by fd limit or memory… (#39689) 
… (#39658)

remove a useless config.
 2024-08-21 15:19:52 +08:00
zhangstar333
1e30d4ebaf Revert "[Bug](compatibility) fix window funnel function coredump when upgrade" (#39681) 
Reverts apache/doris#39646
 2024-08-21 14:47:27 +08:00
meiyi
1460878bdf [fix](cluster key) forbid cluster key and remove case (#39679) 
branch-2.1 does not support mow cluster key
 2024-08-21 14:31:54 +08:00
starocean999
ba3b56d269 [fix](nereids)prevent null pointer exception if datetime value overflows (#39675) 
pick from master https://github.com/apache/doris/pull/39482
 2024-08-21 14:17:34 +08:00
Mingyu Chen
0bfcee1251 [opt](file-cache) support system table file_cache_statistics (#39552) 
1. Add new system table: `file_cache_statistics`

	This table is used for viewing metrics related to file cache on BE side

	```
	mysql> select * from information_schema.file_cache_statistics limit 10;

+-------+---------------+----------------------------+--------------------------------+--------------------+
| BE_ID | BE_IP | CACHE_PATH | METRIC_NAME | METRIC_VALUE |

+-------+---------------+----------------------------+--------------------------------+--------------------+
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_elements | 102400 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
disposable_queue_max_size | 21474836480 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio |
0.8539634687001242 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_1h | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ | hits_ratio_5m | 0
|
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_elements | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_curr_size | 0 |
| 10003 | 172.20.32.136 | /mnt/output/be/file_cache/ |
index_queue_max_elements | 102400 |

+-------+---------------+----------------------------+--------------------------------+--------------------+
	```

	It will show metrics of file caches on each BE.

2. Add new metrics `hits_ratio_1h` and `hits_ratio_5m` for file cache

This 2 metrics will show the hit ratio of file cache in recent 1 hour or
5 minutes.
So that we can know recent hit ratio instead of global historical hit
ratio.
 2024-08-21 10:03:39 +08:00
wuwenchi
bf26f49505 [bugfix](external)add check of engine and catalog types for 2.1 #39343 (#39643) 
bp #39343
 2024-08-21 09:50:17 +08:00
xzj7019
8a562aeb77 [opt](nereids) recover adoptive bucket shuffle (#39598) 
## Proposed changes

pick from https://github.com/apache/doris/pull/36784

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-21 09:26:53 +08:00
wuwenchi
6df6f1dc97 [improvement](iceberg)]support doris's char/varchar to iceberg's string for 2.1 #38807 (#39645) 
bp: #38807
 2024-08-21 09:19:10 +08:00
wuwenchi
28ce116b17 [improvement](iceberg)add some description for show create for 2.1 #39179 (#39644) 
## Proposed changes

bp: #39179

1. add `location` and `properties` for `show create table`.
2. add `location` for `show create database`.
 2024-08-21 09:18:38 +08:00
daidai
27ba2542e2 [case](iceberg)append iceberg schema change case. (#38766) (#39630) 
bp #38766
 2024-08-21 09:17:12 +08:00
yujun
2fe6d580be [improvement](diagnose) add tablet in recycle bin hint #39547 (#39622) 
cherry pick from #39547
 2024-08-21 09:16:01 +08:00
yujun
57262a3d5c [fix](partition rebalancer) fix migrate tablets between backends back and forth #39333 (#39606) 
cherry pick from #39333
 2024-08-21 09:15:31 +08:00
Socrates
bb687bd69c [cherry-pick](branch-2.1) add function regexp_extract_or_null (#39561) 
# Proposed changes

pick https://github.com/apache/doris/pull/38296
 2024-08-21 09:14:58 +08:00
Sun Chenyang
8e9bc7449b [test](inverted index) add test for need read data opt (#38261) (#39534) 
## Proposed changes


pick from master #38261
 2024-08-21 09:01:12 +08:00
zhangstar333
7bb83ae379 [cherry-pick](branch-21) fix append_data_by_selector_impl reserve too mush useless memory (#39581) (#39635) 
## Proposed changes

cherry-pick from master #39581
 2024-08-21 08:47:30 +08:00
zhangstar333
75eded04d7 [Bug](compatibility) fix window funnel function coredump when upgrade (#39646) 
## Proposed changes
this PR https://github.com/apache/doris/pull/39270 have change the agg
of window funnel
and max_be_exec_version is update to 5, in order to compatibility of the
agg function when upgrade.

<!--Describe your changes.-->
 2024-08-21 08:46:50 +08:00
Jibing-Li
7c3c5c67fc [log](statistics)Add result row count log for statistics internal query. (#39556) (#39609) 
backport: https://github.com/apache/doris/pull/39556
 2024-08-20 23:18:48 +08:00
Mingyu Chen
a5daa3edc8 [opt](variables) enlarge the default value of max_allowed_packet (#38697) (#39626) 
bp #38697
 2024-08-20 22:02:01 +08:00
Mingyu Chen
dfd21bd2a0 [fix](fe-log) add position info in async mode #39419 (#39571) 
pick part of #39419
 2024-08-20 22:01:34 +08:00
Mingyu Chen
a4deefea5d [fix](catalog) gen partition id by name (#39325) (#39625) 
bp #39325
 2024-08-20 22:00:19 +08:00
Mingyu Chen
a3fd13fee6 [fix](catalog) set timeout for split fetch (#39346) (#39624) 
bp #39346
 2024-08-20 21:59:55 +08:00
Mingyu Chen
0e21dba817 [opt](catalog) modify some meta cache logic (#38506) (#39628) 
#38506
 2024-08-20 21:57:55 +08:00
Sun Chenyang
12ed2951c4 [fix] (inverted index) remove tmp columns in block (#39369) (#39533) 2024-08-20 20:53:23 +08:00
yujun
607887673e [improvement](report) report handler discard old report tasks #39469 (#39605) 
cherry pick from #39469
 2024-08-20 17:40:49 +08:00
Gabriel
e302882e52 [branch-2.1](pick) Pick 2 PRs to branch-2.1 (#39604) 
## Proposed changes

pick #39480 #39589

<!--Describe your changes.-->
 2024-08-20 17:10:30 +08:00
Xin Liao
5fcd6e6270 [Fix](load) Fix the incorrect src value printed in the error log when strict mode is true #39447 (#39587) 
cherry pick from #39447
 2024-08-20 12:02:13 +08:00
Gavin Chou
621d394a5e [enhance](Backup) Do connectivity check when creating repository (#38350) (#39538) 
Previously when creating repository, FE would not do connectivity check.
It might result in confusing error when using backup restore.

pick #38350

Co-authored-by: AlexYue <yj976240184@gmail.com>
 2024-08-19 22:16:02 +08:00
abmdocrt
3922fdddb6 [cherry-pick](branch-2.1) Pick "[Fix](core) Fix wal mgr heap use after free when stop doris (#33131)" (#39545) 
Pick #33131
 2024-08-19 22:12:09 +08:00
Gavin Chou
85f97a745a [fix](s3) Fix fmt in s3 file wirter S3FileWriter::_dump_completed_part OOM (#39562) 2024-08-19 22:02:06 +08:00
hui lai
9647885b95 [fix](routine load) should update progress before handle transaction state transform (#39311) (#39526) 
pick (#39311)

Update progress maybe throw exception, causing offset has been persisted
on edit log or meta service, but the memory data has not been updated.
It will cause repeated consumption.
 2024-08-19 21:23:59 +08:00
hui lai
3d8b04a782 [fix](stream load) do not throw exception but skip record when can not find database (#39360) (#39527) 
pick (#39360)

When fetch stream load record from BE node, if can not find database,
StreamLoadRecordMgr will throw exception and the remaining records will
not be recorded in memory.

For example: Ten stream load records were pulled, and the database
associated with the stream load of the first record was deleted by the
user. Therefore, the pull will end, resulting in the remaining nine
records not being consumed recorded in memory.

This pr do not throw exception but skip record when can not find
database to solve this problem.
 2024-08-19 21:23:26 +08:00
hui lai
a1aa9b8ab9 [fix](routine load) add read lock to fix some concurrent bugs (#39242) (#39525) 
pick #39242
 2024-08-19 21:18:27 +08:00
Sun Chenyang
fb17f204d7 [fix](http) fix http url with incorrect character notation (#38420) (#39535) 
## Proposed changes

pick from master #38420
 2024-08-19 15:03:19 +08:00
zhiqiang
830f250a80 [opt](query cancel) cancel query if it has pipeline task leakage #39223 (#39537) 
pick #39223 with some modifications. Optimization will only be applied
to pipeline x.
 2024-08-19 14:33:59 +08:00
Siyang Tang
c0cbb2362c [enhancement](schema-change) Record detailed fail reason for schema change tasks (#39351) (#39501) 
## Proposed changes

Expose the error msg from BE as the real fail reason recorded for schema
change tasks. To avoid too much memory usage, we just pick one among all
to record.
 2024-08-18 13:51:06 +08:00
wuwenchi
e01d051acf [improvement](external catalog)Optimize the process of refreshing catalog for 2.1 (#39205) (#39186) 
## Proposed changes

bp: #39205

When the catalog attributes have not changed, refreshing the catalog
only requires processing the cache, without rebuilding the entire
catalog.
 2024-08-17 17:02:06 +08:00
zzzxl
273a62584c [opt](inverted index) unified optimization judgment to prevent omissions (#39473) 
https://github.com/apache/doris/pull/38027
 2024-08-17 16:57:19 +08:00