database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
16,209 Commits 1 Branch 0 Tags
9ef4e49307e9543f83c3b0c68ebc3fd216a1094f
Commit Graph

1220 Commits

Author SHA1 Message Date
yiguolei
9ef4e49307 [bugfix](scannerdeadloop) there is a dead loop in scanner ctx (#29794) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-11 16:47:54 +08:00
Pxl
e556536de1 [Refactor](join) split SetHashTableVariants out from HashTableVariants (#29519) 
split SetHashTableVariants out from HashTableVariants
 2024-01-08 10:37:00 +08:00
yiguolei
c497f749ce [debug](timeout) debug select timeout (#29627) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-07 19:54:02 +08:00
huanghaibin
0b731800a0 [enhancement](group_commit) refector wal manager code (#29560) 2024-01-07 18:54:41 +08:00
Mingyu Chen
2adb0fcc50 [opt](hive) support orc generated from hive 1.x for all file scan node (#28806) 2024-01-06 17:33:16 +08:00
Ashin Gau
5789b7e380 [fix](jin) add datetimev2 precision (#29528) 2024-01-06 13:35:26 +08:00
Qi Chen
8c40f04f2b [Opt](parquet-reader) Opt ColumnSelectVector::set_run_length_null_map() in parquet reader. (#28954) (#29527) 2024-01-05 11:13:40 +08:00
yiguolei
f28dbc702c [bugfix](scanner done) should not set process status to query context (#29512) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-04 15:18:10 +08:00
wuwenchi
bfe65565d8 [feature](paimon)support native reader (#29339) 
Support native reader fro paimon.

Upgrade paimon 0.5 to 0.6 : apache/doris-shade#32
 2024-01-04 14:31:48 +08:00
Pxl
d8a08dad90 [Bug](mark-join) fix wrong result on mark join + other conjunct (#29321) 
fix wrong result on mark join + other conjunct
 2024-01-04 11:58:39 +08:00
yiguolei
bd8113f424 [bugfix](scannerscheduler) should minus num_of_scanners before check should schedule #28926 (#29331) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-03 20:47:35 +08:00
Gabriel
c84cd30223 [pipelineX](fix) Fix query cancel timeout (#29460) 
There are 2 potential reasons to cancel pipelineX query timeout.

Cancel fragment context first and set ready to execute will set cancel flag to false.
Dead lock.
 2024-01-03 20:29:04 +08:00
Jerry Hu
1fbbff32b2 [fix](pipelinex) coredump caused by VRuntimeFilterSlots::_is_global was not set (#29446) 2024-01-03 12:40:41 +08:00
HappenLee
14e7eb7624 [Opt](rf) Opt broadcast join remote runtime filter merge and wait (#29439) 2024-01-03 11:21:28 +08:00
meiyi
706463781c [refactor](group commit) refactor group commit wal code (#29375) 2024-01-02 15:52:03 +08:00
yiguolei
2ed122b787 [improvement](task exec context) add parent class HasTaskExecutionCtx to own the task ctx (#29388) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-02 15:28:27 +08:00
yiguolei
738abac9ed [minor](context) duplicate query context in fragment ctx (#29364) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-01 22:08:23 +08:00
huanghaibin
03901b9a7a [enhancement](group_commit): refector relay wal code (#29183) 2023-12-30 12:59:46 +08:00
Gabriel
b9572f9de0 [pipelineX](fix) Fix pip scanner context bug (#29229) 2023-12-29 13:24:39 +08:00
TengJianPing
a525d5c5a3 [refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion (#29265) 
change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion
 2023-12-29 10:11:44 +08:00
Ashin Gau
a90304c208 [fix](parquet) complex type in parquet is case sensitive (#29245) 
Change name of complex type in parquet to case-insensitive. Otherwise, uppercase column names of complex types will return null.
 2023-12-28 22:43:11 +08:00
Jerry Hu
ffd178f5ff [feat](pipelinex) support parallel scan on pipeline x engine (#29070) 
* [feat](pipelinex) support parallel scan on pipeline x engine

* make parallel scan be independent of shared scan
 2023-12-28 21:29:07 +08:00
Gabriel
8b225c6c3c [pipelineX](fix) Fix core dump if cancelled (#29138) 2023-12-28 10:04:51 +08:00
Gabriel
c75e63a2a5 [Improvement](scan) Use scanner to do projection of scan node (#29124) 2023-12-27 16:00:52 +08:00
Ashin Gau
2d2f14bc75 [fix](paimon) use SlotDescriptor to parse the required fields (#28990) 
Before this PR, Paimon has created the schema of `VectorTable` by accessing meta information. However, once the schema of `VectorTable` in java is not same as `Block` in c++, BE will crashed, and there is no good way to troubleshoot errors.
 2023-12-27 15:45:53 +08:00
Jerry Hu
6440fbfab6 [feature](scan) Implement parallel scanning by dividing the tablets based on the row range (#28967) 
* [feature](scan) parallel scann on dup/mow mode

* fix bugs
 2023-12-26 17:18:41 +08:00
Ashin Gau
f30e50676e [opt](scanner) optimize the number of threads of scanners (#28640) 
1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`
 2023-12-26 10:24:12 +08:00
py023
137f785698 [fix](parquet_reader) misused bool pointer (#28986) 
Signed-off-by: pengyu <pengyu@selectdb.com>
 2023-12-25 22:58:08 +08:00
zhiqiang
c2c5df9341 [opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain (#28935) 
* NEED

* Update pipeline x

* fix pipelinex compile
 2023-12-25 22:47:23 +08:00
TengJianPing
0af9371a96 [fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse (#28991) 
Introduced by #28851, after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block.

e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a.
 2023-12-25 22:19:01 +08:00
caiconghui
7081139bdc [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (#27943) 2023-12-25 20:35:22 +08:00
lihangyu
e9e1e2894b [performance](variant) support topn 2phase read for variant column (#28318) 
[performance](variant) support topn 2phase read for variant column
 2023-12-25 11:50:41 +08:00
yiguolei
1545c36d16 Revert "[bugfix](scannercore) scanner will core in deconstructor during collect profile (#28727)" (#28931) 
This reverts commit 4066de375efe6ff8e156a61df4f9316b3d9eaa4e.
 2023-12-24 20:37:33 +08:00
Kang
db1da161f5 [optimize](zonemap) skip zonemap if predicate does not support_zonemap (#28595) 
* [optimize](zonemap) skip zonemap if predicate does not support_zonemap #27608 (#28506)
 2023-12-24 19:34:13 +08:00
Ashin Gau
96d4778f2e [fix](parquet) the end offset of column chunk may be wrong in parquet metadata (#28891) 2023-12-23 22:21:04 +08:00
yiguolei
4066de375e [bugfix](scannercore) scanner will core in deconstructor during collect profile (#28727) 2023-12-23 11:09:46 +08:00
wangbo
aca8406e31 [refactor](executor)remove scan group #28847 2023-12-22 17:05:50 +08:00
TengJianPing
d75300f166 [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (#28851) 2023-12-22 15:45:12 +08:00
wangbo
012e66729a [improvement](executor) Add tvf and regression test for Workload Scheduler (#28733) 
1 Add select workload schedule policy tvf
2 Add reg test
 2023-12-22 12:09:51 +08:00
huanghaibin
0070909d30 [fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception (#28691) 2023-12-21 20:27:33 +08:00
Yoko
4f1aebb8e8 (topN)runtime_predicate is only triggered when the column name is obtained (#28419) 
Issue Number: close #27485
 2023-12-21 18:08:23 +08:00
Ashin Gau
bcf2683b9d [fix](scanner) fix concurrency bugs when scanner is stopped or finished (#28650) 
`ScannerContext` will schedule scanners even after stopped, and confused with `_is_finished` and `_should_stop`.
 Only Fix the concurrency bugs when scanner is stopped or finished reported in https://github.com/apache/doris/pull/28384
 2023-12-21 10:37:58 +08:00
meiyi
970e1c8475 [fix](group_commit) fix group commit cancel stuck (#28749) 2023-12-21 10:32:21 +08:00
Gabriel
280a01b815 [pipelineX](improvement) Support global runtime filter (#28692) 2023-12-20 20:06:26 +08:00
xy720
504693be7f [bug](coredump) Fix coredump in aggregation node's destruction(#28684) 
fix coredump in aggregation node's destruction
 2023-12-20 20:02:48 +08:00
lihangyu
36857006cd [Fix](json reader) fix json reader crash due to fmt::format_to (#28737) 
```
4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
6# 0x00005622F33D22B1 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
7# 0x00005622F33D2404 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
8# fmt::v7::detail::error_handler::on_error(char const*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
9# char const* fmt::v7::detail::parse_replacement_field<char, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&>(char const*, char const*, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
10# void fmt::v7::detail::vformat_to<char>(fmt::v7::detail::buffer<char>&, fmt::v7::basic_string_view<char>, fmt::v7::basic_format_args<fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<fmt::v7::type_identity<char>::type>, fmt::v7::type_identity<char>::type> >, fmt::v7::detail::locale_ref) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
11# doris::vectorized::NewJsonReader::_append_error_msg(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:924
12# doris::vectorized::NewJsonReader::_set_column_value
```
 2023-12-20 19:58:30 +08:00
zhiqiang
2b2d3d0eb1 [fix](meta_scanner) fix meta_scanner process ColumnNullable (#28711) 2023-12-20 17:41:38 +08:00
wudongliang
111185407c [Improve](tvf)jni-avro support split file (#27933) 2023-12-19 16:37:34 +08:00
yiguolei
b142ade69e [refactor](renamefile) rename some files according to the class names (#28606) 2023-12-19 14:10:11 +08:00
Pxl
89d728290d [Chore](execute) remove some unused code and adjust check_row_nums #28576 2023-12-19 09:55:50 +08:00