a525d5c5a3
[refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion ( #29265 )
...
change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion
2023-12-29 10:11:44 +08:00
a90304c208
[fix](parquet) complex type in parquet is case sensitive ( #29245 )
...
Change name of complex type in parquet to case-insensitive. Otherwise, uppercase column names of complex types will return null.
2023-12-28 22:43:11 +08:00
ffd178f5ff
[feat](pipelinex) support parallel scan on pipeline x engine ( #29070 )
...
* [feat](pipelinex) support parallel scan on pipeline x engine
* make parallel scan be independent of shared scan
2023-12-28 21:29:07 +08:00
8b225c6c3c
[pipelineX](fix) Fix core dump if cancelled ( #29138 )
2023-12-28 10:04:51 +08:00
c75e63a2a5
[Improvement](scan) Use scanner to do projection of scan node ( #29124 )
2023-12-27 16:00:52 +08:00
2d2f14bc75
[fix](paimon) use SlotDescriptor to parse the required fields ( #28990 )
...
Before this PR, Paimon has created the schema of `VectorTable` by accessing meta information. However, once the schema of `VectorTable` in java is not same as `Block` in c++, BE will crashed, and there is no good way to troubleshoot errors.
2023-12-27 15:45:53 +08:00
6440fbfab6
[feature](scan) Implement parallel scanning by dividing the tablets based on the row range ( #28967 )
...
* [feature](scan) parallel scann on dup/mow mode
* fix bugs
2023-12-26 17:18:41 +08:00
f30e50676e
[opt](scanner) optimize the number of threads of scanners ( #28640 )
...
1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`
2023-12-26 10:24:12 +08:00
137f785698
[fix](parquet_reader) misused bool pointer ( #28986 )
...
Signed-off-by: pengyu <pengyu@selectdb.com >
2023-12-25 22:58:08 +08:00
c2c5df9341
[opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain ( #28935 )
...
* NEED
* Update pipeline x
* fix pipelinex compile
2023-12-25 22:47:23 +08:00
0af9371a96
[fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse ( #28991 )
...
Introduced by #28851 , after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block.
e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a.
2023-12-25 22:19:01 +08:00
7081139bdc
[fix](block) fix be core while mutable block merge may cause different row size between columns in origin block ( #27943 )
2023-12-25 20:35:22 +08:00
e9e1e2894b
[performance](variant) support topn 2phase read for variant column ( #28318 )
...
[performance](variant) support topn 2phase read for variant column
2023-12-25 11:50:41 +08:00
1545c36d16
Revert "[bugfix](scannercore) scanner will core in deconstructor during collect profile ( #28727 )" ( #28931 )
...
This reverts commit 4066de375efe6ff8e156a61df4f9316b3d9eaa4e.
2023-12-24 20:37:33 +08:00
db1da161f5
[optimize](zonemap) skip zonemap if predicate does not support_zonemap ( #28595 )
...
* [optimize](zonemap) skip zonemap if predicate does not support_zonemap #27608 (#28506 )
2023-12-24 19:34:13 +08:00
96d4778f2e
[fix](parquet) the end offset of column chunk may be wrong in parquet metadata ( #28891 )
2023-12-23 22:21:04 +08:00
4066de375e
[bugfix](scannercore) scanner will core in deconstructor during collect profile ( #28727 )
2023-12-23 11:09:46 +08:00
aca8406e31
[refactor](executor)remove scan group #28847
2023-12-22 17:05:50 +08:00
d75300f166
[fix](hash join) fix stack overflow caused by evaluate case expr on huge build block ( #28851 )
2023-12-22 15:45:12 +08:00
012e66729a
[improvement](executor) Add tvf and regression test for Workload Scheduler ( #28733 )
...
1 Add select workload schedule policy tvf
2 Add reg test
2023-12-22 12:09:51 +08:00
0070909d30
[fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception ( #28691 )
2023-12-21 20:27:33 +08:00
4f1aebb8e8
(topN)runtime_predicate is only triggered when the column name is obtained ( #28419 )
...
Issue Number: close #27485
2023-12-21 18:08:23 +08:00
bcf2683b9d
[fix](scanner) fix concurrency bugs when scanner is stopped or finished ( #28650 )
...
`ScannerContext` will schedule scanners even after stopped, and confused with `_is_finished` and `_should_stop`.
Only Fix the concurrency bugs when scanner is stopped or finished reported in https://github.com/apache/doris/pull/28384
2023-12-21 10:37:58 +08:00
970e1c8475
[fix](group_commit) fix group commit cancel stuck ( #28749 )
2023-12-21 10:32:21 +08:00
280a01b815
[pipelineX](improvement) Support global runtime filter ( #28692 )
2023-12-20 20:06:26 +08:00
504693be7f
[bug](coredump) Fix coredump in aggregation node's destruction( #28684 )
...
fix coredump in aggregation node's destruction
2023-12-20 20:02:48 +08:00
36857006cd
[Fix](json reader) fix json reader crash due to fmt::format_to ( #28737 )
...
```
4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
6# 0x00005622F33D22B1 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
7# 0x00005622F33D2404 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
8# fmt::v7::detail::error_handler::on_error(char const*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
9# char const* fmt::v7::detail::parse_replacement_field<char, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&>(char const*, char const*, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
10# void fmt::v7::detail::vformat_to<char>(fmt::v7::detail::buffer<char>&, fmt::v7::basic_string_view<char>, fmt::v7::basic_format_args<fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<fmt::v7::type_identity<char>::type>, fmt::v7::type_identity<char>::type> >, fmt::v7::detail::locale_ref) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
11# doris::vectorized::NewJsonReader::_append_error_msg(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:924
12# doris::vectorized::NewJsonReader::_set_column_value
```
2023-12-20 19:58:30 +08:00
2b2d3d0eb1
[fix](meta_scanner) fix meta_scanner process ColumnNullable ( #28711 )
2023-12-20 17:41:38 +08:00
111185407c
[Improve](tvf)jni-avro support split file ( #27933 )
2023-12-19 16:37:34 +08:00
b142ade69e
[refactor](renamefile) rename some files according to the class names ( #28606 )
2023-12-19 14:10:11 +08:00
89d728290d
[Chore](execute) remove some unused code and adjust check_row_nums #28576
2023-12-19 09:55:50 +08:00
66fbb22ad7
[fix](group commit) Fix some wal problems on group commit ( #28554 )
2023-12-19 09:51:03 +08:00
97e63516b7
[fix](streamload) catch exception when reading arrow data ( #28558 )
2023-12-18 22:03:57 +08:00
73f7b61019
[refactor](scanner) use weak ptr to lock task execution context to avoid core in scanner dctor ( #28493 )
...
using weak ptr as a lock between fragment execute thread and scanner thread, to solve the core problem in scanner's dctor to access scannode's profile.
2023-12-18 14:09:32 +08:00
469edbdd3d
[feature](executor)make scan task wait timeout config #28467
2023-12-16 11:36:15 +08:00
fb925bdd08
[Bug](memory) Fix exception-unsafe in aggregation node ( #28483 )
...
The alloc function may throw std::bad_alloc exception when the process memory exceed limit.
be.INFO:
W1214 09:14:17.434849 771103 mem_tracker_limiter.cpp:204] Memory limit exceeded:<consuming tracker:<Load#Id=28448230da1f432e-8a66597e1032
9235>, process memory used 20.41 GB exceed limit 18.76 GB or sys mem available 9.04 GB less than low water mark 1.60 GB, failed alloc siz
e 1.86 MB>, executing msg:<execute:<>>. backend xx.x.x.xxx process memory used 20.41 GB, limit 18.76 GB. If query tracker exceed, set ex ec_mem_limit=8G to change limit, details see be.INFO.
Process Memory Summary:
OS physical memory 31.26 GB. Process memory usage 20.41 GB, limit 18.76 GB, soft limit 16.88 GB. Sys available memory 9.04 GB, low wa
ter mark 1.60 GB, warning water mark 3.20 GB. Refresh interval memory growth 0 B
Alloc Stacktrace:
@ 0x555cd858bee9 doris::MemTrackerLimiter::print_log_usage()
@ 0x555cd859a384 doris::ThreadMemTrackerMgr::exceeded()
@ 0x555cd85a0ac4 malloc
@ 0x555cd8fcf368 Allocator<>::alloc()
@ 0x555cd8fdbdaf doris::vectorized::Arena::add_chunk()
@ 0x555cd96dc0ab doris::vectorized::AggregateDataContainer::_expand()
@ 0x555cd96aded8 (unknown)
@ 0x555cd969fa2c doris::vectorized::AggregationNode::_pre_agg_with_serialized_key()
@ 0x555cd96d1d61 std::_Function_handler<>::_M_invoke()
@ 0x555cd967ab0b doris::vectorized::AggregationNode::get_next()
@ 0x555cd81282a6 doris::ExecNode::get_next_after_projects()
@ 0x555cd8452968 doris::PlanFragmentExecutor::get_vectorized_internal()
@ 0x555cd845553b doris::PlanFragmentExecutor::open_vectorized_internal()
@ 0x555cd8456a9e doris::PlanFragmentExecutor::open()
@ 0x555cd842f200 doris::FragmentExecState::execute()
@ 0x555cd843280e doris::FragmentMgr::_exec_actual()
@ 0x555cd8432d42 _ZNSt17_Function_handlerIFvvEZN5doris11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsESt8funct
ionIFvPNS1_20PlanFragmentExecutorEEEEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x555cd86ead05 doris::ThreadPool::dispatch_thread()
@ 0x555cd86e015f doris::Thread::supervise_thread()
@ 0x7f3321593ea5 start_thread
@ 0x7f33218a69fd __clone
@ (nil) (unknown)
2023-12-15 19:17:18 +08:00
eb99e4270d
[Fix](parquet_reader) Fix dict filtering doesn't work with plain dict encoding in parquet reader. ( #28290 )
2023-12-15 09:27:02 +08:00
9fe2fce306
[minor](refactor) remove unused code ( #28383 )
2023-12-14 17:16:41 +08:00
e53cfa09da
[fix](join) incorrect result of right anti join with nullable ( #28301 )
2023-12-14 14:07:12 +08:00
c00dca70e6
[pipelineX](local shuffle) Support parallel execution despite of tablet number ( #28266 )
2023-12-14 12:53:54 +08:00
48937fef48
[Performance](json reader) optimize filling default values ( #25542 )
...
Add a faster path for filling default values, since looking up value map is relatively slow
2023-12-14 10:20:29 +08:00
ec91dd1129
[opt](vfilescanner) interrupt running parquet/orc readers when scannode is finished ( #28223 )
...
VScanNode::get_next will check whether the ScanNode has reached limit condition, and send eos to TaskScheduler, and TaskScheduler will try to close ScanNode.
However, ScanNode must wait all running scanners finished, so even if ScanNode has reached limit condition, it can't be closed immediately.
This PR try to interrupt the running readers, and make ScanNode to end as soon as possible.
2023-12-13 19:31:08 +08:00
764d893cbf
Remove unused const variables NUMBER, ZERO in vnumbers_tvf.cpp ( #28317 )
...
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com >
2023-12-13 13:16:48 +08:00
13b9350aeb
[Bug](scan)fix some case query timeout of not schedule scanner ( #28243 )
...
now in pipeline, when result block queue is empty, will be reschedule, and then choose a batch of scanner,
but sometimes, get_available_thread_slot_num() will return thread_slot_num <= 0, so it's will do nothing,
and then block queue will always empty.
have no chance to reschedule again until query timeout.
2023-12-12 21:00:22 +08:00
9861cfc4bc
[Fix](Transactional-Hive) Fix transactional hive core dump when TransactionalHiveReader::init_row_filters(). ( #28238 )
...
Fix transactional hive core dump when TransactionalHiveReader::init_row_filters().
2023-12-12 14:17:26 +08:00
5ff110e845
[exec](profile) only build expr debug string enable profile ( #28261 )
2023-12-12 09:13:37 +08:00
ac167f493b
[fix](join) fix decimal overflow caused by left outer join ( #28221 )
...
For left outer join or full outer join, when build side data is empty, null data is output for build side, but nested column data of nullable column is not properly initialized, which may cause decimal arithmetic overflow
2023-12-11 11:51:05 +08:00
e1587537bc
[Fix](status) fix unhandled status in exprs #28218
...
which marked static_cast<void> in https://github.com/apache/doris/pull/23395/files
partially fixed #28160
2023-12-11 11:04:58 +08:00
8f2202c89d
[minor](log) Add debug info in operators ( #28211 )
2023-12-11 10:02:24 +08:00
abc802b5ba
[bugfix](core) child block is shared between operator and node, it should be shared ptr ( #28106 )
...
_child_block in nest loop join , table value function, repeat node will be shared between ExecNode and related operator, but it should not be a unique ptr in operator, it belongs to exec node.
It will double free the block, if operator's close method is not called correctly.
It should be a shared ptr, then it will not core even if the opeartor's close method is not called.
2023-12-09 00:18:14 +08:00
