database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
16,310 Commits 1 Branch 0 Tags
3cf95d0fdfb2fe15d557fcae21e918e209db2dd3
Commit Graph

1227 Commits

Author SHA1 Message Date
Pxl
3cf95d0fdf [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec (#28788) 
optimize for ColumnNullable's serialize_vec/deserialize_vec
 2024-01-12 11:59:52 +08:00
Pxl
33b8311d5f [Improvement](runtime-filter) build runtime filter before build hash table on join build probe (#29727) 
build runtime filter before build hash table on join build probe
 2024-01-12 11:59:52 +08:00
wangbo
0d691c638b [Feature](profile)Support report runtime workload statistics #29591 2024-01-12 11:59:27 +08:00
yiguolei
ca75c9b8ab add more logs to debug timeout 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-12 11:48:39 +08:00
Qi Chen
7287c0ca15 [Opt](exec)(multi-catalog) Opt date type reading. (#29571) 2024-01-12 11:48:39 +08:00
yiguolei
abb7640d37 [debug](timeout) add more log in scanner ctx to find timeout problem #29704 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-12 11:44:21 +08:00
zy-kkk
8fc9c18c85 [improvement](jdbc catalog) Put the jdbc connection pool parameters into catalog properties (#29195) 2024-01-12 11:40:28 +08:00
yiguolei
9ef4e49307 [bugfix](scannerdeadloop) there is a dead loop in scanner ctx (#29794) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-11 16:47:54 +08:00
Pxl
e556536de1 [Refactor](join) split SetHashTableVariants out from HashTableVariants (#29519) 
split SetHashTableVariants out from HashTableVariants
 2024-01-08 10:37:00 +08:00
yiguolei
c497f749ce [debug](timeout) debug select timeout (#29627) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-07 19:54:02 +08:00
huanghaibin
0b731800a0 [enhancement](group_commit) refector wal manager code (#29560) 2024-01-07 18:54:41 +08:00
Mingyu Chen
2adb0fcc50 [opt](hive) support orc generated from hive 1.x for all file scan node (#28806) 2024-01-06 17:33:16 +08:00
Ashin Gau
5789b7e380 [fix](jin) add datetimev2 precision (#29528) 2024-01-06 13:35:26 +08:00
Qi Chen
8c40f04f2b [Opt](parquet-reader) Opt ColumnSelectVector::set_run_length_null_map() in parquet reader. (#28954) (#29527) 2024-01-05 11:13:40 +08:00
yiguolei
f28dbc702c [bugfix](scanner done) should not set process status to query context (#29512) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-04 15:18:10 +08:00
wuwenchi
bfe65565d8 [feature](paimon)support native reader (#29339) 
Support native reader fro paimon.

Upgrade paimon 0.5 to 0.6 : apache/doris-shade#32
 2024-01-04 14:31:48 +08:00
Pxl
d8a08dad90 [Bug](mark-join) fix wrong result on mark join + other conjunct (#29321) 
fix wrong result on mark join + other conjunct
 2024-01-04 11:58:39 +08:00
yiguolei
bd8113f424 [bugfix](scannerscheduler) should minus num_of_scanners before check should schedule #28926 (#29331) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-03 20:47:35 +08:00
Gabriel
c84cd30223 [pipelineX](fix) Fix query cancel timeout (#29460) 
There are 2 potential reasons to cancel pipelineX query timeout.

Cancel fragment context first and set ready to execute will set cancel flag to false.
Dead lock.
 2024-01-03 20:29:04 +08:00
Jerry Hu
1fbbff32b2 [fix](pipelinex) coredump caused by VRuntimeFilterSlots::_is_global was not set (#29446) 2024-01-03 12:40:41 +08:00
HappenLee
14e7eb7624 [Opt](rf) Opt broadcast join remote runtime filter merge and wait (#29439) 2024-01-03 11:21:28 +08:00
meiyi
706463781c [refactor](group commit) refactor group commit wal code (#29375) 2024-01-02 15:52:03 +08:00
yiguolei
2ed122b787 [improvement](task exec context) add parent class HasTaskExecutionCtx to own the task ctx (#29388) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-02 15:28:27 +08:00
yiguolei
738abac9ed [minor](context) duplicate query context in fragment ctx (#29364) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-01 22:08:23 +08:00
huanghaibin
03901b9a7a [enhancement](group_commit): refector relay wal code (#29183) 2023-12-30 12:59:46 +08:00
Gabriel
b9572f9de0 [pipelineX](fix) Fix pip scanner context bug (#29229) 2023-12-29 13:24:39 +08:00
TengJianPing
a525d5c5a3 [refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion (#29265) 
change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion
 2023-12-29 10:11:44 +08:00
Ashin Gau
a90304c208 [fix](parquet) complex type in parquet is case sensitive (#29245) 
Change name of complex type in parquet to case-insensitive. Otherwise, uppercase column names of complex types will return null.
 2023-12-28 22:43:11 +08:00
Jerry Hu
ffd178f5ff [feat](pipelinex) support parallel scan on pipeline x engine (#29070) 
* [feat](pipelinex) support parallel scan on pipeline x engine

* make parallel scan be independent of shared scan
 2023-12-28 21:29:07 +08:00
Gabriel
8b225c6c3c [pipelineX](fix) Fix core dump if cancelled (#29138) 2023-12-28 10:04:51 +08:00
Gabriel
c75e63a2a5 [Improvement](scan) Use scanner to do projection of scan node (#29124) 2023-12-27 16:00:52 +08:00
Ashin Gau
2d2f14bc75 [fix](paimon) use SlotDescriptor to parse the required fields (#28990) 
Before this PR, Paimon has created the schema of `VectorTable` by accessing meta information. However, once the schema of `VectorTable` in java is not same as `Block` in c++, BE will crashed, and there is no good way to troubleshoot errors.
 2023-12-27 15:45:53 +08:00
Jerry Hu
6440fbfab6 [feature](scan) Implement parallel scanning by dividing the tablets based on the row range (#28967) 
* [feature](scan) parallel scann on dup/mow mode

* fix bugs
 2023-12-26 17:18:41 +08:00
Ashin Gau
f30e50676e [opt](scanner) optimize the number of threads of scanners (#28640) 
1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`
 2023-12-26 10:24:12 +08:00
py023
137f785698 [fix](parquet_reader) misused bool pointer (#28986) 
Signed-off-by: pengyu <pengyu@selectdb.com>
 2023-12-25 22:58:08 +08:00
zhiqiang
c2c5df9341 [opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain (#28935) 
* NEED

* Update pipeline x

* fix pipelinex compile
 2023-12-25 22:47:23 +08:00
TengJianPing
0af9371a96 [fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse (#28991) 
Introduced by #28851, after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block.

e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a.
 2023-12-25 22:19:01 +08:00
caiconghui
7081139bdc [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (#27943) 2023-12-25 20:35:22 +08:00
lihangyu
e9e1e2894b [performance](variant) support topn 2phase read for variant column (#28318) 
[performance](variant) support topn 2phase read for variant column
 2023-12-25 11:50:41 +08:00
yiguolei
1545c36d16 Revert "[bugfix](scannercore) scanner will core in deconstructor during collect profile (#28727)" (#28931) 
This reverts commit 4066de375efe6ff8e156a61df4f9316b3d9eaa4e.
 2023-12-24 20:37:33 +08:00
Kang
db1da161f5 [optimize](zonemap) skip zonemap if predicate does not support_zonemap (#28595) 
* [optimize](zonemap) skip zonemap if predicate does not support_zonemap #27608 (#28506)
 2023-12-24 19:34:13 +08:00
Ashin Gau
96d4778f2e [fix](parquet) the end offset of column chunk may be wrong in parquet metadata (#28891) 2023-12-23 22:21:04 +08:00
yiguolei
4066de375e [bugfix](scannercore) scanner will core in deconstructor during collect profile (#28727) 2023-12-23 11:09:46 +08:00
wangbo
aca8406e31 [refactor](executor)remove scan group #28847 2023-12-22 17:05:50 +08:00
TengJianPing
d75300f166 [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28851) 2023-12-22 15:45:12 +08:00
wangbo
012e66729a [improvement](executor) Add tvf and regression test for Workload Scheduler (#28733) 
1 Add select workload schedule policy tvf
2 Add reg test
 2023-12-22 12:09:51 +08:00
huanghaibin
0070909d30 [fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception (#28691) 2023-12-21 20:27:33 +08:00
Yoko
4f1aebb8e8 (topN)runtime_predicate is only triggered when the column name is obtained (#28419) 
Issue Number: close #27485
 2023-12-21 18:08:23 +08:00
Ashin Gau
bcf2683b9d [fix](scanner) fix concurrency bugs when scanner is stopped or finished (#28650) 
`ScannerContext` will schedule scanners even after stopped, and confused with `_is_finished` and `_should_stop`.
 Only Fix the concurrency bugs when scanner is stopped or finished reported in https://github.com/apache/doris/pull/28384
 2023-12-21 10:37:58 +08:00
meiyi
970e1c8475 [fix](group_commit) fix group commit cancel stuck (#28749) 2023-12-21 10:32:21 +08:00