e362bf674f
[regression-test](memtable) test memtable flush is high priority for vtable writerV1 ( #28502 )
2023-12-19 12:33:12 +08:00
d6514618b2
[Improvement](decimal) reduce overhead on disable check decimal overflow ( #28249 )
...
reduce overhead on disable check decimal overflow
2023-12-19 10:12:30 +08:00
89d728290d
[Chore](execute) remove some unused code and adjust check_row_nums #28576
2023-12-19 09:55:50 +08:00
6503aaf7db
[feature](planner) allow HLL and QUANTILE_STATE types on duplicate and unique table ( #28546 )
2023-12-19 09:54:24 +08:00
66fbb22ad7
[fix](group commit) Fix some wal problems on group commit ( #28554 )
2023-12-19 09:51:03 +08:00
97e63516b7
[fix](streamload) catch exception when reading arrow data ( #28558 )
2023-12-18 22:03:57 +08:00
868884145c
[Opt](expr) Opt decimal comparsion expr. ( #28551 )
2023-12-18 18:56:31 +08:00
73f7b61019
[refactor](scanner) use weak ptr to lock task execution context to avoid core in scanner dctor ( #28493 )
...
using weak ptr as a lock between fragment execute thread and scanner thread, to solve the core problem in scanner's dctor to access scannode's profile.
2023-12-18 14:09:32 +08:00
9ebacb1faa
[fix](expr) fix performance problem caused by too many virtual function call ( #28508 )
2023-12-18 12:01:55 +08:00
fbe5a7c244
[improvement](decimalv2) support check overflow for decimalv2 arithmetics ( #28456 )
2023-12-18 10:54:25 +08:00
8ab63a9434
[regression-test][memtable] test memtable flush is high priority for vtable writerV2 ( #28503 )
2023-12-16 22:29:06 +08:00
469edbdd3d
[feature](executor)make scan task wait timeout config #28467
2023-12-16 11:36:15 +08:00
f741ce5b7b
[fix](iterator) Fix mem leak when initial iterator failed ( #28480 )
2023-12-16 10:49:05 +08:00
20d815f0e7
[refactor](style) Using C++style and changing to smart pointers ( #28454 )
2023-12-16 10:44:43 +08:00
fb925bdd08
[Bug](memory) Fix exception-unsafe in aggregation node ( #28483 )
...
The alloc function may throw std::bad_alloc exception when the process memory exceed limit.
be.INFO:
W1214 09:14:17.434849 771103 mem_tracker_limiter.cpp:204] Memory limit exceeded:<consuming tracker:<Load#Id=28448230da1f432e-8a66597e1032
9235>, process memory used 20.41 GB exceed limit 18.76 GB or sys mem available 9.04 GB less than low water mark 1.60 GB, failed alloc siz
e 1.86 MB>, executing msg:<execute:<>>. backend xx.x.x.xxx process memory used 20.41 GB, limit 18.76 GB. If query tracker exceed, set ex ec_mem_limit=8G to change limit, details see be.INFO.
Process Memory Summary:
OS physical memory 31.26 GB. Process memory usage 20.41 GB, limit 18.76 GB, soft limit 16.88 GB. Sys available memory 9.04 GB, low wa
ter mark 1.60 GB, warning water mark 3.20 GB. Refresh interval memory growth 0 B
Alloc Stacktrace:
@ 0x555cd858bee9 doris::MemTrackerLimiter::print_log_usage()
@ 0x555cd859a384 doris::ThreadMemTrackerMgr::exceeded()
@ 0x555cd85a0ac4 malloc
@ 0x555cd8fcf368 Allocator<>::alloc()
@ 0x555cd8fdbdaf doris::vectorized::Arena::add_chunk()
@ 0x555cd96dc0ab doris::vectorized::AggregateDataContainer::_expand()
@ 0x555cd96aded8 (unknown)
@ 0x555cd969fa2c doris::vectorized::AggregationNode::_pre_agg_with_serialized_key()
@ 0x555cd96d1d61 std::_Function_handler<>::_M_invoke()
@ 0x555cd967ab0b doris::vectorized::AggregationNode::get_next()
@ 0x555cd81282a6 doris::ExecNode::get_next_after_projects()
@ 0x555cd8452968 doris::PlanFragmentExecutor::get_vectorized_internal()
@ 0x555cd845553b doris::PlanFragmentExecutor::open_vectorized_internal()
@ 0x555cd8456a9e doris::PlanFragmentExecutor::open()
@ 0x555cd842f200 doris::FragmentExecState::execute()
@ 0x555cd843280e doris::FragmentMgr::_exec_actual()
@ 0x555cd8432d42 _ZNSt17_Function_handlerIFvvEZN5doris11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsESt8funct
ionIFvPNS1_20PlanFragmentExecutorEEEEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x555cd86ead05 doris::ThreadPool::dispatch_thread()
@ 0x555cd86e015f doris::Thread::supervise_thread()
@ 0x7f3321593ea5 start_thread
@ 0x7f33218a69fd __clone
@ (nil) (unknown)
2023-12-15 19:17:18 +08:00
0f25a4b3c6
[bug](json)Fix the problem of be down caused by json path ending with \ ( #28180 )
2023-12-15 15:57:08 +08:00
4d9b6c272d
[Fix](vcompound pred) Corrected evaluation for compound predicates with constant columns ( #28421 )
2023-12-15 10:10:48 +08:00
eb99e4270d
[Fix](parquet_reader) Fix dict filtering doesn't work with plain dict encoding in parquet reader. ( #28290 )
2023-12-15 09:27:02 +08:00
9fe2fce306
[minor](refactor) remove unused code ( #28383 )
2023-12-14 17:16:41 +08:00
e53cfa09da
[fix](join) incorrect result of right anti join with nullable ( #28301 )
2023-12-14 14:07:12 +08:00
c00dca70e6
[pipelineX](local shuffle) Support parallel execution despite of tablet number ( #28266 )
2023-12-14 12:53:54 +08:00
48937fef48
[Performance](json reader) optimize filling default values ( #25542 )
...
Add a faster path for filling default values, since looking up value map is relatively slow
2023-12-14 10:20:29 +08:00
1901f0f35b
[fix](function) bad performance caused by creating Status::RuntimeError ( #28362 )
2023-12-14 09:29:22 +08:00
ec91dd1129
[opt](vfilescanner) interrupt running parquet/orc readers when scannode is finished ( #28223 )
...
VScanNode::get_next will check whether the ScanNode has reached limit condition, and send eos to TaskScheduler, and TaskScheduler will try to close ScanNode.
However, ScanNode must wait all running scanners finished, so even if ScanNode has reached limit condition, it can't be closed immediately.
This PR try to interrupt the running readers, and make ScanNode to end as soon as possible.
2023-12-13 19:31:08 +08:00
9b94faf403
fix](memory_leak) fix memory leak on vrow_distribution #28336
2023-12-13 18:22:14 +08:00
613b0bd552
[fix](decimal) fix decimal overflow caused by null value ( #28260 )
2023-12-13 14:53:39 +08:00
764d893cbf
Remove unused const variables NUMBER, ZERO in vnumbers_tvf.cpp ( #28317 )
...
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com >
2023-12-13 13:16:48 +08:00
78b0fec33a
[Fix](Outfile) Support export nested complex type data to orc file format ( #28182 )
2023-12-13 11:55:27 +08:00
13b9350aeb
[Bug](scan)fix some case query timeout of not schedule scanner ( #28243 )
...
now in pipeline, when result block queue is empty, will be reschedule, and then choose a batch of scanner,
but sometimes, get_available_thread_slot_num() will return thread_slot_num <= 0, so it's will do nothing,
and then block queue will always empty.
have no chance to reschedule again until query timeout.
2023-12-12 21:00:22 +08:00
ace2b45c37
[fix](decimalv2) avoid crashing caused by decimalv2 arithmetic with check_overflow_for_decimal enabled ( #28219 )
2023-12-12 20:32:10 +08:00
45b2dbab6a
[improve](group commit) Group commit support max filter ratio when rows is less than value in config ( #28139 )
2023-12-12 16:33:36 +08:00
9861cfc4bc
[Fix](Transactional-Hive) Fix transactional hive core dump when TransactionalHiveReader::init_row_filters(). ( #28238 )
...
Fix transactional hive core dump when TransactionalHiveReader::init_row_filters().
2023-12-12 14:17:26 +08:00
5ff110e845
[exec](profile) only build expr debug string enable profile ( #28261 )
2023-12-12 09:13:37 +08:00
d4f89389e3
[improve](group commit) Group commit support skip wal ( #27957 )
2023-12-11 19:38:32 +08:00
ac167f493b
[fix](join) fix decimal overflow caused by left outer join ( #28221 )
...
For left outer join or full outer join, when build side data is empty, null data is output for build side, but nested column data of nullable column is not properly initialized, which may cause decimal arithmetic overflow
2023-12-11 11:51:05 +08:00
e1587537bc
[Fix](status) fix unhandled status in exprs #28218
...
which marked static_cast<void> in https://github.com/apache/doris/pull/23395/files
partially fixed #28160
2023-12-11 11:04:58 +08:00
8f2202c89d
[minor](log) Add debug info in operators ( #28211 )
2023-12-11 10:02:24 +08:00
1e5ff40e17
[refactor](group commit) remove future block ( #27720 )
...
Co-authored-by: huanghaibin <284824253@qq.com >
2023-12-11 08:41:51 +08:00
4e86f9bab5
[improve](move-memtable) include and check offset when append data ( #28159 )
2023-12-09 16:21:36 +08:00
363721e066
[Bug](udf) java-udf function open failed cause BE core dump #28063
...
when the java-udf open function failed, and some JNI have not set,
so in close function can't call jni.
2023-12-09 11:00:30 +08:00
027b06059a
[Feature](materialized-view) support count(1) on materialized view ( #28135 )
...
support count(1) on materialized view
fix match failed like select k1, sum(k1) from t group by k1
2023-12-09 01:36:46 +08:00
abc802b5ba
[bugfix](core) child block is shared between operator and node, it should be shared ptr ( #28106 )
...
_child_block in nest loop join , table value function, repeat node will be shared between ExecNode and related operator, but it should not be a unique ptr in operator, it belongs to exec node.
It will double free the block, if operator's close method is not called correctly.
It should be a shared ptr, then it will not core even if the opeartor's close method is not called.
2023-12-09 00:18:14 +08:00
6da36e1077
[feature](merge-cloud) Refactor write path code by abstract base class ( #26537 )
...
Refactor write path code by abstract base class. Whether to use `StorageEngine` or `CloudStorageEngine` will be determined during compilation instead of runtime `config::cloud_mode` to avoid unexpected null pointer or undefined behavior issues caused by merging code.
Class that depend on `StorageEngine` but are shared by the cloud mode need to have an abstract base class. Common code should be extracted into the base class, while the code that depends on `StorageEngine` should be implemented in a `StorageEngine` mix-in class of the base class.
2023-12-08 14:50:36 +08:00
341822ec05
[regression-test](Variant) add compaction case for variant and fix bugs ( #28066 )
2023-12-08 12:18:46 +08:00
d534cdf027
[compile](BE) let arm gcc know some function no return ( #28157 )
...
let arm gcc know some function no return
2023-12-08 11:32:08 +08:00
0947bf4e97
[opt](mysql serde) Avoid core dump when converting invalid block to mysql result ( #28069 )
...
BE will core dump if result block is invalid when we doing result serialization.
An existing bug case is described in #28030 , so we add check branch to avoid BE core dump due to out of range related problem.
2023-12-08 10:21:09 +08:00
25b90eb782
[Feature](function) support random int from specific range ( #28076 )
...
mysql> select rand(-20, -10);
+------------------+
| random(-20, -10) |
+------------------+
| -13 |
+------------------+
1 row in set (0.10 sec)
2023-12-08 10:15:25 +08:00
e75d91c91b
[regression-test](Variant) Add more cases related to schema changes ( #27958 )
...
* [regression-test](Variant) Add more cases related to schema changes
And fix bugs about schema change for variant:
fix bug schema change crash on doing schema change with tablet schema that contains extracted columns
2023-12-08 10:15:12 +08:00
d8d8f15bf3
[improvement](vectorization) Use requires instead of specialization for doris::vectorized::Decimal ( #28027 )
...
Use requires instead of specialization for doris::vectorized::Decimal
2023-12-08 09:59:52 +08:00
9461e86b10
[pipelineX](debug) add debug string ( #28137 )
...
* [pipelineX](debug) add debug string
* update
2023-12-07 23:21:10 +08:00
