afd55123ea
[fix](partial update) report error directly if missing rowsets during doing alignment when flushing memtable due to compaction ( #28677 )
2023-12-20 11:49:11 +08:00
fe184e322a
[code](pipelineX) refine some pipelineX code ( #28570 )
2023-12-20 11:45:06 +08:00
c26c0c31a5
[refactor](runtimefilter) do not use QueryContext in runtime filter ( #28559 )
2023-12-20 10:28:55 +08:00
17268346d3
[Revert](partial update) Revert "Fix missing rowsets during doing alignment when flushing memtable due to compaction ( #28062 )" ( #28674 )
...
This reverts commit 485d7db5161efb011b43a10237df1ca83d5e5c3d.
2023-12-20 10:13:43 +08:00
e59c707022
[fix](stream_load)fix bug for stream ( #27752 )
...
1. forbid thed stream_load without content-length or chunked Transfer Encoding
2. forbid thed stream_load both with content-length and chunked Transfer Encoding
Co-authored-by: xingying01 <xingying01@corp.netease.com >
2023-12-20 09:47:52 +08:00
aab859be56
[enhance](partition_id) check partition id before store meta ( #28055 )
2023-12-19 21:31:41 +08:00
b2d16856b4
[Fix](memtable) fix shrink_memtable_by_agg without duplicated keys ( #28660 )
...
remove duplicated logic:
```
vectorized::Block in_block = _input_mutable_block.to_block();
_put_into_output(in_block);
```
`_input_mutable_block.to_block()` will move `_input_mutable_block`, and lead to `flush` with empty block
2023-12-19 20:45:16 +08:00
9c9249e911
[feature](inverted index) add ignore_above property to prevent long s… ( #28585 )
...
When string is too long, clucene will throw an error.
And the string is too long to analyze. So we ignore the string in index process when the string is longer than 256 bytes by default.
We add an poperty `ignore_above` for user to customize.
2023-12-19 18:54:36 +08:00
c72191eb9e
[refactor](profile&names) using dst_id in pipelinex profile to be same as non pipeline; rename some function names ( #28626 )
...
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-12-19 17:44:29 +08:00
111185407c
[Improve](tvf)jni-avro support split file ( #27933 )
2023-12-19 16:37:34 +08:00
c41562a37b
[Bug](cooldown) Fix problem that followers may never completely cooldown ( #28561 )
2023-12-19 16:03:59 +08:00
95073053bc
[chore] Add bvar for meta operations of BE ( #28374 )
2023-12-19 15:54:19 +08:00
b142ade69e
[refactor](renamefile) rename some files according to the class names ( #28606 )
2023-12-19 14:10:11 +08:00
1253ed006e
[fix](memtable-limiter) do not block write if load mem usage is low ( #28602 )
...
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com >
2023-12-19 13:28:17 +08:00
9434ee5710
[fix](load) fix memtracking orphan too large ( #28600 )
2023-12-19 12:41:19 +08:00
d7dd7b775b
enhance performance for broken tablet checking under multi-core scenario with a coarse-grained read lock ( #28552 )
2023-12-19 12:33:34 +08:00
e362bf674f
[regression-test](memtable) test memtable flush is high priority for vtable writerV1 ( #28502 )
2023-12-19 12:33:12 +08:00
9c17f31767
[regression](memtable) add case for memtable flush error handle ( #28285 )
...
Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn >
2023-12-19 12:32:55 +08:00
7086956b4f
[bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch ( #28597 )
...
* [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch
return RuntimeError if copy_column_data_to_block nullable mismatch to avoid coredump in input_col_ptr->filter_by_selector(sel_rowid_idx, select_size, raw_res_ptr) .
The problem is reported by a doris user but I can not reproduce it, so there is no testcase added currently.
* clang format
2023-12-19 11:30:02 +08:00
5d3eea0ef0
[fix](stacktrace) ignore stacktrace for error code INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED ( #27898 ) ( #28598 )
...
* ignore stacktrace for error INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED
* AndBlockColumnPredicate::evaluate
2023-12-19 11:14:41 +08:00
d17ac99abe
[feature](coverage): refresh the coverage file before exiting the program ( #28354 )
2023-12-19 10:54:57 +08:00
f9ddf8c7ef
[improvement](be report) add be report http ( #28424 )
2023-12-19 10:39:19 +08:00
4163a79669
[pipelineX](profile) make dep time merge ( #28458 )
2023-12-19 10:27:02 +08:00
d6514618b2
[Improvement](decimal) reduce overhead on disable check decimal overflow ( #28249 )
...
reduce overhead on disable check decimal overflow
2023-12-19 10:12:30 +08:00
89d728290d
[Chore](execute) remove some unused code and adjust check_row_nums #28576
2023-12-19 09:55:50 +08:00
6503aaf7db
[feature](planner) allow HLL and QUANTILE_STATE types on duplicate and unique table ( #28546 )
2023-12-19 09:54:24 +08:00
66fbb22ad7
[fix](group commit) Fix some wal problems on group commit ( #28554 )
2023-12-19 09:51:03 +08:00
97e63516b7
[fix](streamload) catch exception when reading arrow data ( #28558 )
2023-12-18 22:03:57 +08:00
868884145c
[Opt](expr) Opt decimal comparsion expr. ( #28551 )
2023-12-18 18:56:31 +08:00
73f7b61019
[refactor](scanner) use weak ptr to lock task execution context to avoid core in scanner dctor ( #28493 )
...
using weak ptr as a lock between fragment execute thread and scanner thread, to solve the core problem in scanner's dctor to access scannode's profile.
2023-12-18 14:09:32 +08:00
9ebacb1faa
[fix](expr) fix performance problem caused by too many virtual function call ( #28508 )
2023-12-18 12:01:55 +08:00
1223f62ce1
[fix](invert index) fix index compaction to write the .tis and .tii file structures ( #28430 )
2023-12-18 11:12:42 +08:00
fbe5a7c244
[improvement](decimalv2) support check overflow for decimalv2 arithmetics ( #28456 )
2023-12-18 10:54:25 +08:00
d11365da9c
[Fix](memtable) fix shrink_memtable_by_agg should also update _row_in_blocks ( #28536 )
...
Otherwise using the stale `_row_in_blocks` will result in heap-buffer-overflow
```
==2695213==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62900122e210 at pc 0x56524744aecf bp 0x7f62c595ef7
0 sp 0x7f62c595ef68
READ of size 8 at 0x62900122e210 thread T1627 (MemTableFlushTh)
#0 0x56524744aece in doris::vectorized::ColumnVector<long>::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_vector.cpp:378:33
#1 0x5652472a7538 in doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_nullable.cpp:310:25
#2 0x56524782a62a in doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/core/block.cpp:961:14
#3 0x565233f187ae in doris::MemTable::_put_into_output(doris::vectorized::Block&) /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:248:27
#4 0x565233f1db66 in doris::MemTable::to_block() /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:496:13
#5 0x565233efae60 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:121:62
#6 0x565233efc8d6 in doris::FlushToken::_flush_memtable(doris::MemTable*, int, long) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:150:16
#7 0x565233f0c5eb in doris::MemtableFlushTask::run() /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:58:23
```
2023-12-18 10:31:16 +08:00
e4585db32d
[enhancement](err-msg) log out datadir path pattern when disk exceed capacity limit #28320
2023-12-16 23:07:02 +08:00
61de49c727
[case](regression) Test duplicated load id ( #28251 )
...
Co-authored-by: qinhao <qinhao@newland.com.cn >
2023-12-16 22:41:51 +08:00
894bae4ebf
[improvement](publish version) publish txn fail retry do not wait ( #28441 )
2023-12-16 22:31:10 +08:00
8ab63a9434
[regression-test][memtable] test memtable flush is high priority for vtable writerV2 ( #28503 )
2023-12-16 22:29:06 +08:00
ad8faedac4
[fix](txn_manager) Fix wrong use of std::map::erase in TxnManager::delete_txn #28507
2023-12-16 14:50:46 +08:00
b11b76e778
[fix](full compaction) Full compaction should hold meta lock when modifying tablet's meta data ( #28449 )
2023-12-16 12:37:29 +08:00
469edbdd3d
[feature](executor)make scan task wait timeout config #28467
2023-12-16 11:36:15 +08:00
4538f1ba8f
[feature](pipelineX) add local_shuffle in nested loop join #28428
2023-12-16 10:53:13 +08:00
f741ce5b7b
[fix](iterator) Fix mem leak when initial iterator failed ( #28480 )
2023-12-16 10:49:05 +08:00
f770403cca
[enhancement](pipeline) add bvar for pipeline fragment instance and task ( #28500 )
2023-12-16 10:47:53 +08:00
20d815f0e7
[refactor](style) Using C++style and changing to smart pointers ( #28454 )
2023-12-16 10:44:43 +08:00
fb925bdd08
[Bug](memory) Fix exception-unsafe in aggregation node ( #28483 )
...
The alloc function may throw std::bad_alloc exception when the process memory exceed limit.
be.INFO:
W1214 09:14:17.434849 771103 mem_tracker_limiter.cpp:204] Memory limit exceeded:<consuming tracker:<Load#Id=28448230da1f432e-8a66597e1032
9235>, process memory used 20.41 GB exceed limit 18.76 GB or sys mem available 9.04 GB less than low water mark 1.60 GB, failed alloc siz
e 1.86 MB>, executing msg:<execute:<>>. backend xx.x.x.xxx process memory used 20.41 GB, limit 18.76 GB. If query tracker exceed, set ex ec_mem_limit=8G to change limit, details see be.INFO.
Process Memory Summary:
OS physical memory 31.26 GB. Process memory usage 20.41 GB, limit 18.76 GB, soft limit 16.88 GB. Sys available memory 9.04 GB, low wa
ter mark 1.60 GB, warning water mark 3.20 GB. Refresh interval memory growth 0 B
Alloc Stacktrace:
@ 0x555cd858bee9 doris::MemTrackerLimiter::print_log_usage()
@ 0x555cd859a384 doris::ThreadMemTrackerMgr::exceeded()
@ 0x555cd85a0ac4 malloc
@ 0x555cd8fcf368 Allocator<>::alloc()
@ 0x555cd8fdbdaf doris::vectorized::Arena::add_chunk()
@ 0x555cd96dc0ab doris::vectorized::AggregateDataContainer::_expand()
@ 0x555cd96aded8 (unknown)
@ 0x555cd969fa2c doris::vectorized::AggregationNode::_pre_agg_with_serialized_key()
@ 0x555cd96d1d61 std::_Function_handler<>::_M_invoke()
@ 0x555cd967ab0b doris::vectorized::AggregationNode::get_next()
@ 0x555cd81282a6 doris::ExecNode::get_next_after_projects()
@ 0x555cd8452968 doris::PlanFragmentExecutor::get_vectorized_internal()
@ 0x555cd845553b doris::PlanFragmentExecutor::open_vectorized_internal()
@ 0x555cd8456a9e doris::PlanFragmentExecutor::open()
@ 0x555cd842f200 doris::FragmentExecState::execute()
@ 0x555cd843280e doris::FragmentMgr::_exec_actual()
@ 0x555cd8432d42 _ZNSt17_Function_handlerIFvvEZN5doris11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsESt8funct
ionIFvPNS1_20PlanFragmentExecutorEEEEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x555cd86ead05 doris::ThreadPool::dispatch_thread()
@ 0x555cd86e015f doris::Thread::supervise_thread()
@ 0x7f3321593ea5 start_thread
@ 0x7f33218a69fd __clone
@ (nil) (unknown)
2023-12-15 19:17:18 +08:00
0f25a4b3c6
[bug](json)Fix the problem of be down caused by json path ending with \ ( #28180 )
2023-12-15 15:57:08 +08:00
4d9b6c272d
[Fix](vcompound pred) Corrected evaluation for compound predicates with constant columns ( #28421 )
2023-12-15 10:10:48 +08:00
eb99e4270d
[Fix](parquet_reader) Fix dict filtering doesn't work with plain dict encoding in parquet reader. ( #28290 )
2023-12-15 09:27:02 +08:00
eebedbc879
[optimize](cooldown)Reduce unnecessary sort operations for vector ( #27147 )
...
Co-authored-by: xingying01 <xingying01@corp.netease.com >
2023-12-15 00:13:56 +08:00
