database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
16,432 Commits 1 Branch 0 Tags
779ed259724377f5e8d9556fb504f32bdc07681e
Commit Graph

1238 Commits

Author SHA1 Message Date
Jerry Hu
779ed25972 [fix](scan) crashing caused by unlocked reading of tablet (#30005) 2024-01-16 18:46:19 +08:00
nanfeng
be893d792c [fix](jni) fix jni_reader function name get_nex_block to get_next_block (#29943) 2024-01-16 18:39:00 +08:00
wangbo
e35b26f4fc [feature](auditlog)Add runtime cpu time/peak memory metric (#29925) 2024-01-16 18:39:00 +08:00
Jerry Hu
b7b8e59392 [opt](scanner) use buffered queue to avoid acquiring locks frequently (#29938) 2024-01-16 18:37:44 +08:00
Jerry Hu
c8845c9e07 [opt](scanner) Improve the efficiency of TOPN opt (#29937) 2024-01-16 18:37:44 +08:00
HHoflittlefish777
615d94bbc7 [log](insertadd log in parse insert into values data (#29903) 2024-01-16 18:35:32 +08:00
yiguolei
5e697990a8 [bugfix](timeout) serving_blocks_num may cause timeout, try to fix it (#29912) 
Although serving_blocks_num is an atomic variable. It's ++ and -- are not protected by transfer lock.
I am not sure the memory order of ++ and --.
I think it maybe the root cause of query timeout. So that I remove the check and test it in github pipeline.
 2024-01-16 18:34:19 +08:00
yiguolei
e4e57e9b05 [chore](removelogs) remove debug query timeout logs 2024-01-12 14:37:20 +08:00
Xujian Duan
ad2c13e009 [Optimize](kill-query)Support the scanners exits as soon as possible when kill query #29803 2024-01-12 13:58:19 +08:00
Qi Chen
d494674ff4 [opt](parquet-reader) Opt parquet decimal type reading. (#29825) 2024-01-12 13:58:19 +08:00
Jerry Hu
f02fb5d49e [fix](vec) wrong implementation of operator <=> of Field (#29743) 2024-01-12 11:59:52 +08:00
Pxl
3cf95d0fdf [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec (#28788) 
optimize for ColumnNullable's serialize_vec/deserialize_vec
 2024-01-12 11:59:52 +08:00
Pxl
33b8311d5f [Improvement](runtime-filter) build runtime filter before build hash table on join build probe (#29727) 
build runtime filter before build hash table on join build probe
 2024-01-12 11:59:52 +08:00
wangbo
0d691c638b [Feature](profile)Support report runtime workload statistics #29591 2024-01-12 11:59:27 +08:00
yiguolei
ca75c9b8ab add more logs to debug timeout 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-12 11:48:39 +08:00
Qi Chen
7287c0ca15 [Opt](exec)(multi-catalog) Opt date type reading. (#29571) 2024-01-12 11:48:39 +08:00
yiguolei
abb7640d37 [debug](timeout) add more log in scanner ctx to find timeout problem #29704 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-12 11:44:21 +08:00
zy-kkk
8fc9c18c85 [improvement](jdbc catalog) Put the jdbc connection pool parameters into catalog properties (#29195) 2024-01-12 11:40:28 +08:00
yiguolei
9ef4e49307 [bugfix](scannerdeadloop) there is a dead loop in scanner ctx (#29794) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-11 16:47:54 +08:00
Pxl
e556536de1 [Refactor](join) split SetHashTableVariants out from HashTableVariants (#29519) 
split SetHashTableVariants out from HashTableVariants
 2024-01-08 10:37:00 +08:00
yiguolei
c497f749ce [debug](timeout) debug select timeout (#29627) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-07 19:54:02 +08:00
huanghaibin
0b731800a0 [enhancement](group_commit) refector wal manager code (#29560) 2024-01-07 18:54:41 +08:00
Mingyu Chen
2adb0fcc50 [opt](hive) support orc generated from hive 1.x for all file scan node (#28806) 2024-01-06 17:33:16 +08:00
Ashin Gau
5789b7e380 [fix](jin) add datetimev2 precision (#29528) 2024-01-06 13:35:26 +08:00
Qi Chen
8c40f04f2b [Opt](parquet-reader) Opt ColumnSelectVector::set_run_length_null_map() in parquet reader. (#28954) (#29527) 2024-01-05 11:13:40 +08:00
yiguolei
f28dbc702c [bugfix](scanner done) should not set process status to query context (#29512) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-04 15:18:10 +08:00
wuwenchi
bfe65565d8 [feature](paimon)support native reader (#29339) 
Support native reader fro paimon.

Upgrade paimon 0.5 to 0.6 : apache/doris-shade#32
 2024-01-04 14:31:48 +08:00
Pxl
d8a08dad90 [Bug](mark-join) fix wrong result on mark join + other conjunct (#29321) 
fix wrong result on mark join + other conjunct
 2024-01-04 11:58:39 +08:00
yiguolei
bd8113f424 [bugfix](scannerscheduler) should minus num_of_scanners before check should schedule #28926 (#29331) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-03 20:47:35 +08:00
Gabriel
c84cd30223 [pipelineX](fix) Fix query cancel timeout (#29460) 
There are 2 potential reasons to cancel pipelineX query timeout.

Cancel fragment context first and set ready to execute will set cancel flag to false.
Dead lock.
 2024-01-03 20:29:04 +08:00
Jerry Hu
1fbbff32b2 [fix](pipelinex) coredump caused by VRuntimeFilterSlots::_is_global was not set (#29446) 2024-01-03 12:40:41 +08:00
HappenLee
14e7eb7624 [Opt](rf) Opt broadcast join remote runtime filter merge and wait (#29439) 2024-01-03 11:21:28 +08:00
meiyi
706463781c [refactor](group commit) refactor group commit wal code (#29375) 2024-01-02 15:52:03 +08:00
yiguolei
2ed122b787 [improvement](task exec context) add parent class HasTaskExecutionCtx to own the task ctx (#29388) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-02 15:28:27 +08:00
yiguolei
738abac9ed [minor](context) duplicate query context in fragment ctx (#29364) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-01 22:08:23 +08:00
huanghaibin
03901b9a7a [enhancement](group_commit): refector relay wal code (#29183) 2023-12-30 12:59:46 +08:00
Gabriel
b9572f9de0 [pipelineX](fix) Fix pip scanner context bug (#29229) 2023-12-29 13:24:39 +08:00
TengJianPing
a525d5c5a3 [refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion (#29265) 
change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion
 2023-12-29 10:11:44 +08:00
Ashin Gau
a90304c208 [fix](parquet) complex type in parquet is case sensitive (#29245) 
Change name of complex type in parquet to case-insensitive. Otherwise, uppercase column names of complex types will return null.
 2023-12-28 22:43:11 +08:00
Jerry Hu
ffd178f5ff [feat](pipelinex) support parallel scan on pipeline x engine (#29070) 
* [feat](pipelinex) support parallel scan on pipeline x engine

* make parallel scan be independent of shared scan
 2023-12-28 21:29:07 +08:00
Gabriel
8b225c6c3c [pipelineX](fix) Fix core dump if cancelled (#29138) 2023-12-28 10:04:51 +08:00
Gabriel
c75e63a2a5 [Improvement](scan) Use scanner to do projection of scan node (#29124) 2023-12-27 16:00:52 +08:00
Ashin Gau
2d2f14bc75 [fix](paimon) use SlotDescriptor to parse the required fields (#28990) 
Before this PR, Paimon has created the schema of `VectorTable` by accessing meta information. However, once the schema of `VectorTable` in java is not same as `Block` in c++, BE will crashed, and there is no good way to troubleshoot errors.
 2023-12-27 15:45:53 +08:00
Jerry Hu
6440fbfab6 [feature](scan) Implement parallel scanning by dividing the tablets based on the row range (#28967) 
* [feature](scan) parallel scann on dup/mow mode

* fix bugs
 2023-12-26 17:18:41 +08:00
Ashin Gau
f30e50676e [opt](scanner) optimize the number of threads of scanners (#28640) 
1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`
 2023-12-26 10:24:12 +08:00
py023
137f785698 [fix](parquet_reader) misused bool pointer (#28986) 
Signed-off-by: pengyu <pengyu@selectdb.com>
 2023-12-25 22:58:08 +08:00
zhiqiang
c2c5df9341 [opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain (#28935) 
* NEED

* Update pipeline x

* fix pipelinex compile
 2023-12-25 22:47:23 +08:00
TengJianPing
0af9371a96 [fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse (#28991) 
Introduced by #28851, after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block.

e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a.
 2023-12-25 22:19:01 +08:00
caiconghui
7081139bdc [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (#27943) 2023-12-25 20:35:22 +08:00
lihangyu
e9e1e2894b [performance](variant) support topn 2phase read for variant column (#28318) 
[performance](variant) support topn 2phase read for variant column
 2023-12-25 11:50:41 +08:00