database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
16,865 Commits 1 Branch 0 Tags
4b42156fc0d3b7828fb70f5f591c4c5facd508eb
Commit Graph

466 Commits

Author SHA1 Message Date
py023
4b42156fc0 [chore](clang-tidy): add bugprone linters (#29521) 
This PR introduces 4 bugprone linter rules to .clang-tidy, these linters found some bugs in #28965. This PR also add some comments to mute false positive reports.
 2024-02-05 21:58:08 +08:00
Qi Chen
6289f7e605 [Fix](multi-catalog) Fix truncate_char_or_varchar_column crash. (#30731) 2024-02-03 20:26:04 +08:00
zy-kkk
4f8730d092 [improvement](jdbc catalog) Optimize connection pool parameter settings (#30588) 
This PR makes the following changes to the connection pool of JDBC Catalog
1. Set the maximum connection survival time, the default is 30 minutes

-   Moreover, one-half of the maximum survival time is the recyclable time,
-   One-tenth is the check interval for recycling connections

2. Keepalive only takes effect on the connection pool on BE, and will be activated based on one-fifth of the maximum survival time.
3. The maximum number of existing connections is changed from 100 to 10
4. Add the connection cache recycling thread on BE, and add a parameter to control the recycling time, the default is 28800 (8 hours)
5. Add CatalogID to the key of the connection pool cache to achieve better isolation, requires refresh catalog to take effect
6. Upgrade druid connection pool to version 1.2.20
7. Added JdbcResource's setting of default parameters when upgrading the FE version to avoid errors due to unset parameters.
 2024-02-03 20:26:03 +08:00
Gabriel
045225a096 [pipelineX](profile) Fix Tablet counter on pipelineX engine (#30613) 2024-01-31 23:53:39 +08:00
wuwenchi
7d037c12bf [bugfix](paimon)fix paimon testcases (#30514) 
1. set default timezone
2. not supported `char` type to pushdown
 2024-01-31 23:53:39 +08:00
HappenLee
378d9e7336 [Colo][Scan] delete the colo scan code (#30584) 2024-01-31 23:53:39 +08:00
HappenLee
129463f557 [Try_Fix](scan) try fix the scanner schedule logic to prevent excessive memory usage and timeout (#30515) 2024-01-30 15:31:22 +08:00
Yongqiang YANG
bedad15f03 [enhancement](scanner) add a lower bound for bytes in scanner queue (#29624) 2024-01-27 09:13:21 +08:00
wangbo
9e0c518aaf [Feature](executor)Workload Group support Non-Pipeline Execution (#30164) 2024-01-23 10:11:25 +08:00
yiguolei
d3bf23d70d [chore](removelogs) remove debug query timeout logs (#30006) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-16 18:48:18 +08:00
Jerry Hu
779ed25972 [fix](scan) crashing caused by unlocked reading of tablet (#30005) 2024-01-16 18:46:19 +08:00
wangbo
e35b26f4fc [feature](auditlog)Add runtime cpu time/peak memory metric (#29925) 2024-01-16 18:39:00 +08:00
Jerry Hu
b7b8e59392 [opt](scanner) use buffered queue to avoid acquiring locks frequently (#29938) 2024-01-16 18:37:44 +08:00
Jerry Hu
c8845c9e07 [opt](scanner) Improve the efficiency of TOPN opt (#29937) 2024-01-16 18:37:44 +08:00
yiguolei
5e697990a8 [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it (#29912) 
Although serving_blocks_num is an atomic variable. It's ++ and -- are not protected by transfer lock.
I am not sure the memory order of ++ and --.
I think it maybe the root cause of query timeout. So that I remove the check and test it in github pipeline.
 2024-01-16 18:34:19 +08:00
yiguolei
e4e57e9b05 [chore](removelogs) remove debug query timeout logs 2024-01-12 14:37:20 +08:00
Xujian Duan
ad2c13e009 [Optimize](kill-query)Support the scanners exits as soon as possible when kill query #29803 2024-01-12 13:58:19 +08:00
wangbo
0d691c638b [Feature](profile)Support report runtime workload statistics #29591 2024-01-12 11:59:27 +08:00
yiguolei
ca75c9b8ab add more logs to debug timeout 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-12 11:48:39 +08:00
yiguolei
abb7640d37 [debug](timeout) add more log in scanner ctx to find timeout problem #29704 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-12 11:44:21 +08:00
zy-kkk
8fc9c18c85 [improvement](jdbc catalog) Put the jdbc connection pool parameters into catalog properties (#29195) 2024-01-12 11:40:28 +08:00
yiguolei
9ef4e49307 [bugfix](scannerdeadloop) there is a dead loop in scanner ctx (#29794) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-11 16:47:54 +08:00
yiguolei
c497f749ce [debug](timeout) debug select timeout (#29627) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-07 19:54:02 +08:00
huanghaibin
0b731800a0 [enhancement](group_commit) refector wal manager code (#29560) 2024-01-07 18:54:41 +08:00
yiguolei
f28dbc702c [bugfix](scanner done) should not set process status to query context (#29512) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-04 15:18:10 +08:00
wuwenchi
bfe65565d8 [feature](paimon)support native reader (#29339) 
Support native reader fro paimon.

Upgrade paimon 0.5 to 0.6 : apache/doris-shade#32
 2024-01-04 14:31:48 +08:00
yiguolei
bd8113f424 [bugfix](scannerscheduler) should minus num_of_scanners before check should schedule #28926 (#29331) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-03 20:47:35 +08:00
Gabriel
c84cd30223 [pipelineX](fix) Fix query cancel timeout (#29460) 
There are 2 potential reasons to cancel pipelineX query timeout.

Cancel fragment context first and set ready to execute will set cancel flag to false.
Dead lock.
 2024-01-03 20:29:04 +08:00
yiguolei
2ed122b787 [improvement](task exec context) add parent class HasTaskExecutionCtx to own the task ctx (#29388) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-02 15:28:27 +08:00
yiguolei
738abac9ed [minor](context) duplicate query context in fragment ctx (#29364) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-01 22:08:23 +08:00
Gabriel
b9572f9de0 [pipelineX](fix) Fix pip scanner context bug (#29229) 2023-12-29 13:24:39 +08:00
Jerry Hu
ffd178f5ff [feat](pipelinex) support parallel scan on pipeline x engine (#29070) 
* [feat](pipelinex) support parallel scan on pipeline x engine

* make parallel scan be independent of shared scan
 2023-12-28 21:29:07 +08:00
Gabriel
c75e63a2a5 [Improvement](scan) Use scanner to do projection of scan node (#29124) 2023-12-27 16:00:52 +08:00
Jerry Hu
6440fbfab6 [feature](scan) Implement parallel scanning by dividing the tablets based on the row range (#28967) 
* [feature](scan) parallel scann on dup/mow mode

* fix bugs
 2023-12-26 17:18:41 +08:00
Ashin Gau
f30e50676e [opt](scanner) optimize the number of threads of scanners (#28640) 
1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`
 2023-12-26 10:24:12 +08:00
caiconghui
7081139bdc [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (#27943) 2023-12-25 20:35:22 +08:00
lihangyu
e9e1e2894b [performance](variant) support topn 2phase read for variant column (#28318) 
[performance](variant) support topn 2phase read for variant column
 2023-12-25 11:50:41 +08:00
yiguolei
1545c36d16 Revert "[bugfix](scannercore) scanner will core in deconstructor during collect profile (#28727)" (#28931) 
This reverts commit 4066de375efe6ff8e156a61df4f9316b3d9eaa4e.
 2023-12-24 20:37:33 +08:00
Kang
db1da161f5 [optimize](zonemap) skip zonemap if predicate does not support_zonemap (#28595) 
* [optimize](zonemap) skip zonemap if predicate does not support_zonemap #27608 (#28506)
 2023-12-24 19:34:13 +08:00
yiguolei
4066de375e [bugfix](scannercore) scanner will core in deconstructor during collect profile (#28727) 2023-12-23 11:09:46 +08:00
wangbo
aca8406e31 [refactor](executor)remove scan group #28847 2023-12-22 17:05:50 +08:00
wangbo
012e66729a [improvement](executor) Add tvf and regression test for Workload Scheduler (#28733) 
1 Add select workload schedule policy tvf
2 Add reg test
 2023-12-22 12:09:51 +08:00
Ashin Gau
bcf2683b9d [fix](scanner) fix concurrency bugs when scanner is stopped or finished (#28650) 
`ScannerContext` will schedule scanners even after stopped, and confused with `_is_finished` and `_should_stop`.
 Only Fix the concurrency bugs when scanner is stopped or finished reported in https://github.com/apache/doris/pull/28384
 2023-12-21 10:37:58 +08:00
meiyi
970e1c8475 [fix](group_commit) fix group commit cancel stuck (#28749) 2023-12-21 10:32:21 +08:00
zhiqiang
2b2d3d0eb1 [fix](meta_scanner) fix meta_scanner process ColumnNullable (#28711) 2023-12-20 17:41:38 +08:00
yiguolei
b142ade69e [refactor](renamefile) rename some files according to the class names (#28606) 2023-12-19 14:10:11 +08:00
huanghaibin
66fbb22ad7 [fix](group commit) Fix some wal problems on group commit (#28554) 2023-12-19 09:51:03 +08:00
yiguolei
73f7b61019 [refactor](scanner) use weak ptr to lock task execution context to avoid core in scanner dctor (#28493) 
using weak ptr as a lock between fragment execute thread and scanner thread, to solve the core problem in scanner's dctor to access scannode's profile.
 2023-12-18 14:09:32 +08:00
wangbo
469edbdd3d [feature](executor)make scan task wait timeout config #28467 2023-12-16 11:36:15 +08:00
Gabriel
9fe2fce306 [minor](refactor) remove unused code (#28383) 2023-12-14 17:16:41 +08:00