pick #51217
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
…(#50938)
https://github.com/apache/doris/pull/50938
if cast return st not ok
```
report error status: [E-254] std::bad_typeid
0# doris::get_stack_trace(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/util/stack_util.cpp:59
1# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/common/exception.cpp:30
2# doris::vectorized::ColumnStr<unsigned int> const* typeid_cast<doris::vectorized::ColumnStr<unsigned int> const*, doris::vectorized::IColumn const>(doris::vectorized::IColumn const*) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/common/typeid_cast.h:42
3# doris::vectorized::ColumnStr<unsigned int> const* doris::vectorized::check_and_get_column<doris::vectorized::ColumnStr<unsigned int> >(doris::vectorized::IColumn const*) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/columns/column.h:725
4# doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/exprs/vexpr_context.cpp:61
```
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Problem Summary:
pick from master (#50913)
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Pick #50993
Introduced by #51050
The build block maybe be `clear_column_mem_not_keep` in build phase when
the operator is closed.
```cpp
Status HashJoinBuildSinkLocalState::close(RuntimeState* state, Status exec_status) {
if (_closed) {
return Status::OK();
}
auto& p = _parent->cast<HashJoinBuildSinkOperatorX>();
Defer defer {[&]() {
if (!_should_build_hash_table) {
return;
}
// The build side hash key column maybe no need output, but we need to keep the column in block
// because it is used to compare with probe side hash key column
if (p._should_keep_hash_key_column && _build_col_ids.size() == 1) {
p._should_keep_column_flags[_build_col_ids[0]] = true;
}
if (_shared_state->build_block) {
// release the memory of unused column in probe stage
_shared_state->build_block->clear_column_mem_not_keep(p._should_keep_column_flags,
p._use_shared_hash_table);
}
if (p._use_shared_hash_table) {
std::unique_lock lock(p._mutex);
p._signaled = true;
for (auto& dep : _shared_state->sink_deps) {
dep->set_ready();
}
for (auto& dep : p._finish_dependencies) {
dep->set_ready();
}
}
}};
```
```
*** Aborted at 1747343165 (unix time) try "date -d @1747343165" if you are using GNU date ***
*** Current BE git commitID: e7a3e78b97 ***
*** SIGSEGV address not mapped to object (@0x1) received by PID 7474 (TID 9641 OR 0x7f3f8c0e5640) from PID 1; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
3# 0x00007F4368F76520 in /lib/x86_64-linux-gnu/libc.so.6
4# doris::Status doris::pipeline::ProcessHashTableProbe<7>::finish_probing > > >(doris::vectorized::MethodKeysFixed > >&, doris::vectorized::MutableBlock&, doris::vectorized::Block*, bool*, bool) at /root/doris/be/src/pipeline/exec/join/process_hash_table_probe_impl.h:738
5# std::__detail::__variant::__gen_vtable_impl (*)(doris::pipeline::HashJoinProbeOperatorX::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) const::$_1&&, std::variant > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber, doris::JoinHashTable, HashCRC32 > > >, doris::vectorized::MethodOneNumber, doris::JoinHashTable, HashCRC32 > > >, doris::vectorized::MethodKeysFixed > >, doris::vectorized::MethodKeysFixed, HashCRC32 > > >, doris::vectorized::MethodKeysFixed > >, doris::vectorized::MethodKeysFixed, HashCRC32 > > >, doris::vectorized::MethodStringNoCache > > >&, std::variant, doris::pipeline::ProcessHashTableProbe<2>, doris::pipeline::ProcessHashTableProbe<8>, doris::pipeline::ProcessHashTableProbe<1>, doris::pipeline::ProcessHashTableProbe<4>, doris::pipeline::ProcessHashTableProbe<3>, doris::pipeline::ProcessHashTableProbe<7>, doris::pipeline::ProcessHashTableProbe<9>, doris::pipeline::ProcessHashTableProbe<10>, doris::pipeline::ProcessHashTableProbe<11> >&)>, std::integer_sequence >::__visit_invoke(doris::pipeline::HashJoinProbeOperatorX::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) const::$_1&&, std::variant > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber, doris::JoinHashTable, HashCRC32 > > >, doris::vectorized::MethodOneNumber, doris::JoinHashTable, HashCRC32 > > >, doris::vectorized::MethodKeysFixed > >, doris::vectorized::MethodKeysFixed, HashCRC32 > > >, doris::vectorized::MethodKeysFixed > >, doris::vectorized::MethodKeysFixed, HashCRC32 > > >, doris::vectorized::MethodStringNoCache > > >&, std::variant, doris::pipeline::ProcessHashTableProbe<2>, doris::pipeline::ProcessHashTableProbe<8>, doris::pipeline::ProcessHashTableProbe<1>, doris::pipeline::ProcessHashTableProbe<4>, doris::pipeline::ProcessHashTableProbe<3>, doris::pipeline::ProcessHashTableProbe<7>, doris::pipeline::ProcessHashTableProbe<9>, doris::pipeline::ProcessHashTableProbe<10>, doris::pipeline::ProcessHashTableProbe<11> >&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013
6# doris::pipeline::HashJoinProbeOperatorX::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) const at /root/doris/be/src/pipeline/exec/hashjoin_probe_operator.cpp:281
7# doris::pipeline::StatefulOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/pipeline/exec/operator.cpp:670
8# doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/pipeline/exec/operator.cpp:381
9# doris::pipeline::PipelineTask::execute(bool*) in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be
10# doris::pipeline::TaskScheduler::_do_work(int) at /root/doris/be/src/pipeline/task_scheduler.cpp:144
11# doris::ThreadPool::dispatch_thread() at /root/doris/be/src/util/threadpool.cpp:622
12# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:469
13# start_thread at ./nptl/pthread_create.c:442
14# 0x00007F436905A850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```
Related PR: #xxx
Problem Summary:
None
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
pick #48963
Part IV of https://github.com/apache/doris/issues/48511
doc https://github.com/apache/doris-website/pull/2196
**Introduce routine load job statistic system table:**
```
mysql> show create table information_schema.routine_load_job\G
*************************** 1. row ***************************
Table: routine_load_job
Create Table: CREATE TABLE `routine_load_job` (
`JOB_ID` text NULL,
`JOB_NAME` text NULL,
`CREATE_TIME` text NULL,
`PAUSE_TIME` text NULL,
`END_TIME` text NULL,
`DB_NAME` text NULL,
`TABLE_NAME` text NULL,
`STATE` text NULL,
`CURRENT_TASK_NUM` text NULL,
`JOB_PROPERTIES` text NULL,
`DATA_SOURCE_PROPERTIES` text NULL,
`CUSTOM_PROPERTIES` text NULL,
`STATISTIC` text NULL,
`PROGRESS` text NULL,
`LAG` text NULL,
`REASON_OF_STATE_CHANGED` text NULL,
`ERROR_LOG_URLS` text NULL,
`USER_NAME` text NULL,
`CURRENT_ABORT_TASK_NUM` int NULL,
`IS_ABNORMAL_PAUSE` boolean NULL
) ENGINE=SCHEMA;
1 row in set (0.00 sec)
```
**There are some benefits to empower job with SQL query capability for
statistical information:**
- It can be used in conjunction with metrics add through
https://github.com/apache/doris/pull/48209 to roughly locate abnormal
jobs when Grafana alarms, and the following SQL can be used:
```
SELECT JOB_NAME
FROM information_schema.routine_load_job_statistics
WHERE CURRENT_ABORT_TASK_NUM > 0
OR IS_ABNORMAL_PAUSE = TRUE;
```
- User can use the `select * from information_schema.routine_load_job`
instead of the `show routine load`. The advantage is that the `show
routine load` can only be searched by name, but SQL can be very flexible
in locating jobs
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
* Add `-XX:+PrintClassHistogramAfterFullGC` for JAVA_OPTS
* Add `classhisto*=trace` for JAVA_OPTS_FOR_JDK_17
fe.gc.log will print like this:
```
2024-11-15T11:49:00.316+0800: 11.346: [Class Histogram (after full gc):
num #instances #bytes class name
----------------------------------------------
1: 7464 7053464 [B
2: 37465 3656360 [C
3: 7076 2909880 [Ljava.lang.Object;
4: 4915 2306872 [I
5: 9167 1719552 [S
6: 16229 1168488 io.grpc.netty.shaded.io.netty.buffer.PoolSubpage
......
```
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [x] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
