Introduced by #45207.
For a query plan,
```
set sink operator -- -|
|-> set source operator
set probe operator --|
```
If `set source operators` are finished (due to limit reached), all
upstream operators could be finished and waken up at the same time.
However, some states are initialized in `set sink operator`. So if `set
probe operator` executes before `set sink operator` initialization, it
will incur an invalid address access.
*** Query id: cebb723bbda64249-9ab8c9e7aa72c540 *** *** is nereids: 1
***
*** tablet id: 0 ***
*** Aborted at 1736092087 (unix time) try "date -d @1736092087" if you
are using GNU date ***
*** Current BE git commitID: 26d68d778a ***
*** SIGSEGV address not mapped to object (@0xf8) received by PID 23579
(TID 26524 OR 0x7f84d4240640) from PID 248; stack trace: *** 0#
doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2#
JVM_handle_linux_signal in
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
3# 0x00007F88F4063520 in /lib/x86_64-linux-gnu/libc.so.6
4#
doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&)
at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:183
5# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*,
doris::vectorized::Block*, bool) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:99
6# doris::pipeline::PipelineTask::execute(bool*) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:383
7# doris::pipeline::TaskScheduler::_do_work(int) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/task_scheduler.cpp:138
8# doris::ThreadPool::dispatch_thread() in
/mnt/hdd01/ci/master-deploy/be/lib/doris_be
9# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 10#
start_thread at ./nptl/pthread_create.c:442
11# 0x00007F88F4147850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
cherry-pick #38653
Issue Number: close #xxx
F20240731 10:38:53.012670 20850 mem_tracker_limiter.cpp:135] mem tracker
label: Query#Id=7539c7d0087442b7-a1cda6392053548a, consumption: 52, peak
consumption: 1155332, mem tracker not equal to 0 when mem tracker
destruct, this usually means that memory tracking is inaccurate and
SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not
used correctly. 1. For query and load, memory leaks may have occurred,
it is expected that the query mem tracker will be bound to the thread
context using SCOPED_ATTACH_TASK and
SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER before all memory alloc and
free. 2. If a memory alloc is recorded by this tracker, it is expected
that be recorded in this tracker when memory is freed. 3. Merge the
remaining memory tracking value by this tracker into Orphan, if you
observe that Orphan is not equal to 0 in the mem tracker web or log,
this indicates that there may be a memory leak. 4. If you need to
transfer memory tracking value between two trackers, can use
transfer_to..[Address Sanitizer]:
0x606002867d80, size 52, strack trace:
0# Allocator::alloc_impl(unsigned long, unsigned long) 1# void*
phmap::priv::Allocate<4ul, doris::vectorized::Allocator_
>(doris::vectorized::Allocator_*, unsigned long)
2# phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::initialize_slots(unsigned long) 3#
phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::resize(unsigned long) 4#
phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::prepare_insert(unsigned long) 5#
std::pair phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::find_or_prepare_insert(int const&,
unsigned long)
6# std::pair, phmap::Hash, phmap::EqualTo, doris::vectorized::Allocator_
>::iterator, bool> phmap::priv::raw_hash_set, phmap::Hash,
phmap::EqualTo, doris::vectorized::Allocator_
>::emplace_decomposable(int const&, unsigned long, int const&)
7# std::pair, phmap::Hash, phmap::EqualTo, doris::vectorized::Allocator_
>::iterator, bool> phmap::priv::raw_hash_set, phmap::Hash,
phmap::EqualTo, doris::vectorized::Allocator_
>::EmplaceDecomposable::operator()(int const&, int const&) const
8# doris::HybridSet<(doris::PrimitiveType)5, doris::DynamicContainer,
doris::vectorized::ColumnVector >::insert(void const*) 9#
doris::HybridSetBase::insert(doris::HybridSetBase*) 10#
doris::RuntimePredicateWrapper::merge(doris::RuntimePredicateWrapper
const*)
11# doris::IRuntimeFilter::merge_from(doris::RuntimePredicateWrapper
const*)
12#
doris::RuntimeFilterMergeControllerEntity::merge(doris::PMergeFilterRequest
const*, butil::IOBufAsZeroCopyInputStream*)
13# doris::FragmentMgr::merge_filter(doris::PMergeFilterRequest const*,
butil::IOBufAsZeroCopyInputStream*)
14# std::_Function_handler::_M_invoke(std::_Any_data const&) 15#
doris::WorkThreadPool::work_thread(int)
16# execute_native_thread_routine
17# ?
18# ?
if we use ipv6_cidr_to_range function with nullable func which with
invalid ipv6 will make be core
```
mysql> select id, ipv6_cidr_to_range(nullable(''), 32) from fn_test_ip_nullable order by id;
```
In the past, there were issues with converting `double` and `decimal` to
`boolean`.
For example, a `double` value like 0.13 would first be cast to `uint8`,
resulting in 0.
Then, it would be converted to `bool`, yielding 0 (incorrect, as the
expected result is 1).
Similarly, `decimal` values were directly cast to `uint8`, leading to
non-0/1 values for `bool`.
This issue arises because Doris internally uses `uint8` to represent
`boolean`.
before
```
mysql> select cast(40.123 as BOOLEAN);
+-------------------------+
| cast(40.123 as BOOLEAN) |
+-------------------------+
| 40 |
+-------------------------+
```
now
```
mysql> select cast(40.123 as BOOLEAN);
+-------------------------+
| cast(40.123 as BOOLEAN) |
+-------------------------+
| 1 |
+-------------------------+
```
### What problem does this PR solve?
pick #44326
Related PR: #[44326](https://github.com/mrhhsg/doris/tree/pick_44326)
Problem Summary:
cherry-pick #45414
before：
```
mysqlslap -hd3 -uroot -P9130 --create-schema=test_db2 -c 10 -i 500 -q "SELECT count(k) FROM sbtest1_dup WHERE k BETWEEN 4850578 AND 8454295 OR k BETWEEN 8776291 AND 29749077;"
Benchmark
Average number of seconds to run all queries: 0.041 seconds
Minimum number of seconds to run all queries: 0.037 seconds
Maximum number of seconds to run all queries: 0.115 seconds
Number of clients running queries: 10
Average number of queries per client: 1
```
after：
```
mysqlslap -hd3 -uroot -P9030 --create-schema=test_db -c 10 -i 500 -q "SELECT count(k) FROM sbtest1 WHERE k BETWEEN 4850578 AND 8454295 OR k BETWEEN 8776291 AND 29749077;"
Benchmark
Average number of seconds to run all queries: 0.029 seconds
Minimum number of seconds to run all queries: 0.027 seconds
Maximum number of seconds to run all queries: 0.034 seconds
Number of clients running queries: 10
Average number of queries per client: 1
```
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
pick #44907
In production, we encountered an issue where the librdkafka consumer
stucked during destruction, causing the heavy work pool to become
saturated, which in turn made all heavy work pool-dependent
functionalities, such as querying, unusable. To mitigate this impact, we
replaced the heavy work pool with routine load threads for metadata
fetching.
…ck when PrimitiveType to PColumnType (#39985)
use exception to replace dcheck when PrimitiveType to PColumnType
```cpp
*** SIGABRT unknown detail explain (@0x11d3f) received by PID 73023 (TID 74292 OR 0x7fd758225640) from PID 73023; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# 0x00007FDDBE6B9520 in /lib/x86_64-linux-gnu/libc.so.6
2# pthread_kill at ./nptl/pthread_kill.c:89
3# raise at ../sysdeps/posix/raise.c:27
4# abort at ./stdlib/abort.c:81
5# 0x000056123F81A94D in /root/output/be/lib/doris_be
6# 0x000056123F80CF8A in /root/output/be/lib/doris_be
7# google::LogMessage::SendToLog() in /root/output/be/lib/doris_be
8# google::LogMessage::Flush() in /root/output/be/lib/doris_be
9# google::LogMessageFatal::~LogMessageFatal() in /root/output/be/lib/doris_be
10# doris::to_proto(doris::PrimitiveType) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:114
11# doris::IRuntimeFilter::push_to_remote(doris::TNetworkAddress const*) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:1143
12# doris::IRuntimeFilter::publish(bool)::$_0::operator()(doris::IRuntimeFilter*) const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:959
13# doris::IRuntimeFilter::publish(bool)::$_2::operator()() const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:983
14# doris::IRuntimeFilter::publish(bool) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:997
```
## Proposed changes
pick from #39985
Before this PR, in cases where there is an alternating distribution of
data rowset -> delete rowset -> data rowset -> delete rowset, cumulative
compaction would only move the cumulative point forward to allow base
compaction to handle the delete rowset. Cumulative compaction itself
would not process the data and would return be marked as failure. This
would cause the compaction submission task process to pause for 5
seconds, impacting efficiency.
This PR modifies the return status to OK for such cases, which improves
the efficiency of the compaction submission task.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #43185
Pick the pr to branch-2.1 to fix predicate filter failed when use hive
1.x version
Co-authored-by: fantasy12345zsq <1575033031@qq.com>
