a6f267c479
[pick](Variant) fix element_at should return nullable if result type is nullable ( #39846 )
...
#39732
2024-08-24 09:22:03 +08:00
8cf6c6a2b5
[fix](agg function) incorrect result of map agg( #39743 ) ( #39854 )
...
## Proposed changes
pick #39743
2024-08-24 09:19:52 +08:00
3103bb08dc
[pick](Variant) casting to decimal type may lost precision ( #39843 )
...
#39650
2024-08-23 22:47:32 +08:00
37443aa7e1
[improve](move-memtable) reuse connection in load_stream_stub ( #39231 ) ( #39762 )
...
backport #39231
2024-08-23 22:46:28 +08:00
6ceb574aa0
[branch-2.1]Pick IO limit/workload group usage table ( #39839 )
2024-08-23 18:51:47 +08:00
baf5b71b39
[branch-2.1](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache ( #39829 )
...
pick #38185
2024-08-23 17:21:42 +08:00
c40246efa9
[bugfix](iceberg)Fixed random core with writing iceberg partitioned table for 2.1 ( #39808 )( #39569 ) ( #39832 )
...
## Proposed changes
bp: #39808 #39569
2024-08-23 17:19:48 +08:00
1f16daa5f6
Revert "[bugfix](iceberg)clear block for partition values for 2.1 ( #39569 )" ( #39815 )
...
Reverts apache/doris#39729
2024-08-23 11:58:42 +08:00
1367f74e7a
[branch-2.1](memory) Optimize ClearCacheActionimplementation ( #39796 )
...
pick #38438
2024-08-23 01:51:14 +08:00
dc732fe33f
[bugfix](iceberg)clear block for partition values for 2.1 ( #39569 ) ( #39729 )
...
## Proposed changes
bp: #39569
clear block, or we will get wrong partition values.
2024-08-22 22:43:02 +08:00
04e993c1de
[refine](pipeline) refine some VDataStreamRecvr code ( #35063 ) ( #37802 )
...
## Proposed changes
https://github.com/apache/doris/pull/35063
https://github.com/apache/doris/pull/35428
2024-08-22 19:55:17 +08:00
8ce8887b75
[branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth ( #39760 )
...
pick #38168
overwrites changes in #37221 on workload_group_manager.cpp. If need to
pick 37221, ignore it.
2024-08-22 17:33:11 +08:00
ba7baa7e6b
[fix](window_funnel) fix upgrading problem caused by behaviour change of window_funnel ( #39766 )
...
## Proposed changes
Issue Number: close #xxx
For the latest published 2.1 version `2.1.5`:
```
max_be_exec_version=4;
AGG_FUNCTION_NEW=2;
```
and `branch-2.1`:
```
max_be_exec_version=5;
AGG_FUNCTION_NEW=2;
```
It will cause problem when upgrading.
This PR fix the problem, set `AGG_FUNCTION_NEW` to `5`.
2024-08-22 17:26:51 +08:00
0e694f19db
[fix](merge-on-write) segcompaction should process delete bitmap if necessary ( #38369 ) ( #39707 )
...
## Proposed changes
Issue Number: close #xxx
cherry-pick #38369 and #38800
2024-08-22 00:42:56 +08:00
1e30d4ebaf
Revert "[Bug](compatibility) fix window funnel function coredump when upgrade" ( #39681 )
...
Reverts apache/doris#39646
2024-08-21 14:47:27 +08:00
bb687bd69c
[cherry-pick](branch-2.1) add function regexp_extract_or_null ( #39561 )
...
# Proposed changes
pick https://github.com/apache/doris/pull/38296
2024-08-21 09:14:58 +08:00
7bb83ae379
[cherry-pick](branch-21) fix append_data_by_selector_impl reserve too mush useless memory ( #39581 ) ( #39635 )
...
## Proposed changes
cherry-pick from master #39581
2024-08-21 08:47:30 +08:00
75eded04d7
[Bug](compatibility) fix window funnel function coredump when upgrade ( #39646 )
...
## Proposed changes
this PR https://github.com/apache/doris/pull/39270 have change the agg
of window funnel
and max_be_exec_version is update to 5, in order to compatibility of the
agg function when upgrade.
<!--Describe your changes.-->
2024-08-21 08:46:50 +08:00
a3fd13fee6
[fix](catalog) set timeout for split fetch ( #39346 ) ( #39624 )
...
bp #39346
2024-08-20 21:59:55 +08:00
12ed2951c4
[fix] (inverted index) remove tmp columns in block ( #39369 ) ( #39533 )
2024-08-20 20:53:23 +08:00
5fcd6e6270
[Fix](load) Fix the incorrect src value printed in the error log when strict mode is true #39447 ( #39587 )
...
cherry pick from #39447
2024-08-20 12:02:13 +08:00
273a62584c
[opt](inverted index) unified optimization judgment to prevent omissions ( #39473 )
...
https://github.com/apache/doris/pull/38027
2024-08-17 16:57:19 +08:00
b0da8430bc
[opt](inverted index) Optimize the usage of the multi_match function ( #39472 )
...
## Proposed changes
https://github.com/apache/doris/pull/39193
<!--Describe your changes.-->
2024-08-17 16:53:52 +08:00
79aa079cc6
[fix](array-funcs) array min/max #39307 ( #39484 )
2024-08-17 10:56:44 +08:00
7687f2c53a
[fix](ip-funcs) fix ip inet6_aton funcs #39415 ( #39513 )
2024-08-17 10:56:06 +08:00
824f035b98
[pick](Row store) fix row store with invalid json string in variant ty… ( #39456 )
...
#39394
2024-08-16 14:43:11 +08:00
021678c7c3
[fix](window_funnel) fix wrong result of window_funnel #38954 ( #39270 )
...
## Proposed changes
BP #38954
2024-08-16 09:59:31 +08:00
a44a274563
[Fix](parquet-reader) Fix and optimize parquet min-max filtering. ( #39375 )
...
Backport #38277 .
2024-08-15 14:12:54 +08:00
acf07cab6f
[refactor](minor) Init counter in prepare phase ( #39287 ) ( #39385 )
...
pick #39287
2024-08-15 13:36:12 +08:00
226e01889c
[fix](array_apply) pick array apply fix ( #39328 )
...
## Proposed changes
backport: https://github.com/apache/doris/pull/39105
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-14 18:52:29 +08:00
78d6e318fb
[fix](ip)pick ip rowstore ( #39345 )
...
## Proposed changes
backport: https://github.com/apache/doris/pull/39258
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-14 18:51:58 +08:00
b26af32934
[fix](function) fix error return type in corr(float32,float32) ( #39251 ) ( #39350 )
...
https://github.com/apache/doris/pull/39251
```
mysql [test11]>select corr(cast(x as float),cast(y as float)) from test_corr;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]column_type not match data_types in agg node, column_type=Nullable(Float64), data_types=Nullable(Float32),column name=
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-14 18:47:14 +08:00
677435cef8
[Pick](Branch-2.1) pick json reader fix and support specify $. as column ( #39271 )
...
#39206
#38213
2024-08-13 17:44:45 +08:00
7e7729c4b0
[cherry-pick](branch-21) fix partition-topn calculate partition input rows have error ( #39100 ) ( #39281 )
...
## Proposed changes
cherry-pick from master: #39100
<!--Describe your changes.-->
2024-08-13 17:42:29 +08:00
b976cbf14d
[opt](log) avoid lots of json parse error logs ( #39190 ) ( #39246 )
...
pick https://github.com/apache/doris/pull/39190 to branch-2.1
2024-08-13 17:01:10 +08:00
60eeec3754
[fix] (inverted index) Fix match function without inverted index ( #38989 ) ( #39220 )
...
## Proposed changes
pick from #38989
2024-08-13 10:55:54 +08:00
a6155a517d
[fix] (topn) fix uncleared block in topn_next() ( #39119 ) ( #39224 )
...
## Proposed changes
pick from master #39119
2024-08-13 10:34:17 +08:00
96260d97bd
[fix](ub) undefined behavior in FixedContainer ( #39191 ) ( #39201 )
...
## Proposed changes
pick #39191
Undefined behavior occurs if there is a null value in the list.
```
/root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null
/var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here
#0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41
#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60
#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36
#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63
#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21
#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26
#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19
#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5
#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9
#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12
#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38
#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5
#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9
#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5
#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5
#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25
#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9
#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19
#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33
#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29
#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h
#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24
#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14
#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27
#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9
#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5
#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19
#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32
#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8
#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-12 10:02:07 +08:00
b38caed808
[Improve](columns)replace fatal with exception #38035 ( #38996 )
2024-08-12 09:51:30 +08:00
3da2d1c9d6
[bug](parquet)Fix the problem that the parquet reader reads the missing sub-columns of the struct and fails. ( #38718 ) ( #39192 )
...
bp #38718
2024-08-11 20:37:40 +08:00
5f77f909d9
[cherry-pick](branch-2.1) Pick "[feature](function) support ip functions named ipv4_to_ipv6 and cut_ipv6" ( #39058 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
pick https://github.com/apache/doris/pull/36883 and
https://github.com/apache/doris/pull/35239
2024-08-10 18:37:11 +08:00
5e1e725cee
[feature](inverted index) Add multi_match function #37722 #38931 #39149 ( #38877 )
2024-08-10 15:20:08 +08:00
e15b6cfc68
[fix](be) return correct canceled status from scanner ( #36392 ) ( #39111 )
...
## Proposed changes
pick #36392
2024-08-09 04:02:42 +08:00
0571342538
[fix](sink) The issue with 2GB limit of protocol buffer ( #37990 ) ( #39112 )
...
```
Fail to serialize doris.PFetchDataResult
```
If the size of `PFetchDataResult` is greater than 2G, protocol buffer
cannot serialize the message.
pick #37990
2024-08-09 04:01:56 +08:00
efdd75f286
[fix](function) stddev with DecimalV2 type will result in an error (#… ( #39072 )
...
…38731)
https://github.com/apache/doris/pull/38731
The stddev function has a separate implementation for the DecimalV2
type, but there are issues with the implementation. Given that there is
almost no existing data for DecimalV2, it will be removed here. For be,
upgrading to this situation will result in an error directly.
```
SELECT STDDEV(data) FROM DECIMALV2_10_0_DATA;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Agg Function stddev(decimal(10,0)) is not implemented
```
After removing DecimalV2, parameters of type DecimalV2 will be converted
to double for calculations.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-08 17:53:17 +08:00
44cb7978a9
[opt](index) add more inverted index profile metrics #36696 ( #38858 )
2024-08-08 14:16:55 +08:00
0a3874f203
[fix](move-memtable) close stream when cancel load stream stub ( #38912 ) ( #39039 )
...
backport #38912
2024-08-07 23:24:00 +08:00
773008d6fa
[Fix](Json) fix some cast issue ( #38683 ) ( #39025 )
...
#38683
2024-08-07 22:05:43 +08:00
7550fbaff7
[Fix](Exception) throw exception in defer may result std::terminate (… ( #39007 )
...
pick #38935
2024-08-07 13:46:23 +08:00
8cb5aa64f4
[test](inverted index) add an Inverted Index Testing Switch ( #38077 ) ( #38947 )
...
https://github.com/apache/doris/pull/38077
2024-08-07 11:25:36 +08:00
