Commit Graph

8885 Commits

Author SHA1 Message Date
56d87c7f4d [cherry-pick](branch-21) fix array_map cause coredump as NULL (#51618) (#51742) 2025-06-16 14:53:43 +08:00
3c85740a1e branch-2.1: [Opt](metric) Add metrics for invalid delete bitmap key count (#51654)
### What problem does this PR solve?

```
# TYPE doris_be_invalid_delete_bitmap_key_count gauge
doris_be_invalid_delete_bitmap_key_count 0
# TYPE doris_be_valid_delete_bitmap_key_count gauge
doris_be_valid_delete_bitmap_key_count 409
```
https://github.com/apache/doris/pull/50080
2025-06-13 11:24:54 +08:00
18d2f93120 branch-2.1: [fix](function) JSON_EXTRACT_STRING should return NULL instead of the string 'null' when encountering a NULL value #51516 (#51566)
Cherry-picked from #51516

---------

Co-authored-by: Jerry Hu <hushenggang@selectdb.com>
2025-06-13 11:07:31 +08:00
a836df5d53 branch-2.1: [fix](decimal)Fix the issue where decimal multiplication produces incorrect results due to mul_overflow error #51533 (#51563)
Cherry-picked from #51533

Co-authored-by: Mryange <2319153948@qq.com>
2025-06-07 20:28:29 +08:00
c8daf2e4fb branch-2.1: [fix](cache prune) Fix _stale_sweep_time_s not working when its value less than 0 #42281 (#51189)
Cherry-picked from #42281

Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
2025-06-05 12:14:24 +08:00
14cb4c90ad [Fix](Load) fix load hang when the instance cancel with memory tracker before exe… (#51428)
…cution

when the load task cancelled before execution as memory limitation, the
load task will hang as the coordinator is waiting the report from BE.

solution:
Let BE report the task status when task cancelled before execution .


Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-06-04 20:47:29 +08:00
92fa037e5a branch-2.1: [enhancement](memoryfailed) throw exception if block.serialize meet error status #51335 (#51365)
Cherry-picked from #51335

Co-authored-by: Pxl <xl@selectdb.com>
2025-05-30 09:35:28 +08:00
42f4498c9d branch-2.1: [fix](iceberg) fix the iceberg eq-delete filter resize-fill bug. #51253 (#51316)
Cherry-picked from #51253

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
2025-05-28 22:38:00 +08:00
91abfe38cd [Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. (#51276)
### What problem does this PR solve?

Problem Summary:
2025-05-28 14:30:27 +08:00
dd89a78d30 branch-2.1:[fix](parquet/orc) Disable string dictionary filtering when predicate express is not binary pred and in pred (#50749) (#51266)
cherry-pick: #50749
2025-05-27 18:12:56 +08:00
9713c7a1e5 [cherry-pick](branch-2.1) Pick "[Fix](case) Fix cold data compaction fault injection case (#49699)" (#51249) 2025-05-27 17:50:33 +08:00
67e400483d [branch-2.1 fix](expr) fix expr set result column id even when state is not ok #50938 (#51225)
…(#50938)
https://github.com/apache/doris/pull/50938
if cast return st not ok
```
report error status: [E-254] std::bad_typeid
	0#  doris::get_stack_trace(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/util/stack_util.cpp:59
	1#  doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/common/exception.cpp:30
	2#  doris::vectorized::ColumnStr<unsigned int> const* typeid_cast<doris::vectorized::ColumnStr<unsigned int> const*, doris::vectorized::IColumn const>(doris::vectorized::IColumn const*) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/common/typeid_cast.h:42
	3#  doris::vectorized::ColumnStr<unsigned int> const* doris::vectorized::check_and_get_column<doris::vectorized::ColumnStr<unsigned int> >(doris::vectorized::IColumn const*) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/columns/column.h:725
	4#  doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) at /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/exprs/vexpr_context.cpp:61

```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-26 18:05:39 +08:00
df464f84b1 branch-2.1: [fix](load) fix the error msg of task submission failure for memory back pressure #51078 (#51131)
Cherry-picked from #51078

Co-authored-by: hui lai <laihui@selectdb.com>
2025-05-22 20:37:39 +08:00
fbad523a13 [cherry-pick](branch-21) pick (#50913) (#51072)
### What problem does this PR solve?
Problem Summary:
pick from master (#50913)

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-22 14:34:29 +08:00
974263d83c [fix](join) Should not use the build block's size to resize mark_join_flags (#50993) (#51089)
Pick #50993

Introduced by #51050

The build block maybe be `clear_column_mem_not_keep` in build phase when
the operator is closed.

```cpp
Status HashJoinBuildSinkLocalState::close(RuntimeState* state, Status exec_status) {
    if (_closed) {
        return Status::OK();
    }
    auto& p = _parent->cast<HashJoinBuildSinkOperatorX>();
    Defer defer {[&]() {
        if (!_should_build_hash_table) {
            return;
        }
        // The build side hash key column maybe no need output, but we need to keep the column in block
        // because it is used to compare with probe side hash key column

        if (p._should_keep_hash_key_column && _build_col_ids.size() == 1) {
            p._should_keep_column_flags[_build_col_ids[0]] = true;
        }

        if (_shared_state->build_block) {
            // release the memory of unused column in probe stage
            _shared_state->build_block->clear_column_mem_not_keep(p._should_keep_column_flags,
                                                                  p._use_shared_hash_table);
        }

        if (p._use_shared_hash_table) {
            std::unique_lock lock(p._mutex);
            p._signaled = true;
            for (auto& dep : _shared_state->sink_deps) {
                dep->set_ready();
            }
            for (auto& dep : p._finish_dependencies) {
                dep->set_ready();
            }
        }
    }};
```

```
*** Aborted at 1747343165 (unix time) try "date -d @1747343165" if you are using GNU date ***
*** Current BE git commitID: e7a3e78b97 ***
*** SIGSEGV address not mapped to object (@0x1) received by PID 7474 (TID 9641 OR 0x7f3f8c0e5640) from PID 1; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:421
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F4368F76520 in /lib/x86_64-linux-gnu/libc.so.6
 4# doris::Status doris::pipeline::ProcessHashTableProbe<7>::finish_probing > > >(doris::vectorized::MethodKeysFixed > >&, doris::vectorized::MutableBlock&, doris::vectorized::Block*, bool*, bool) at /root/doris/be/src/pipeline/exec/join/process_hash_table_probe_impl.h:738
 5# std::__detail::__variant::__gen_vtable_impl (*)(doris::pipeline::HashJoinProbeOperatorX::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) const::$_1&&, std::variant > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber, doris::JoinHashTable, HashCRC32 > > >, doris::vectorized::MethodOneNumber, doris::JoinHashTable, HashCRC32 > > >, doris::vectorized::MethodKeysFixed > >, doris::vectorized::MethodKeysFixed, HashCRC32 > > >, doris::vectorized::MethodKeysFixed > >, doris::vectorized::MethodKeysFixed, HashCRC32 > > >, doris::vectorized::MethodStringNoCache > > >&, std::variant, doris::pipeline::ProcessHashTableProbe<2>, doris::pipeline::ProcessHashTableProbe<8>, doris::pipeline::ProcessHashTableProbe<1>, doris::pipeline::ProcessHashTableProbe<4>, doris::pipeline::ProcessHashTableProbe<3>, doris::pipeline::ProcessHashTableProbe<7>, doris::pipeline::ProcessHashTableProbe<9>, doris::pipeline::ProcessHashTableProbe<10>, doris::pipeline::ProcessHashTableProbe<11> >&)>, std::integer_sequence >::__visit_invoke(doris::pipeline::HashJoinProbeOperatorX::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) const::$_1&&, std::variant > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber, doris::JoinHashTable, HashCRC32 > > >, doris::vectorized::MethodOneNumber, doris::JoinHashTable, HashCRC32 > > >, doris::vectorized::MethodKeysFixed > >, doris::vectorized::MethodKeysFixed, HashCRC32 > > >, doris::vectorized::MethodKeysFixed > >, doris::vectorized::MethodKeysFixed, HashCRC32 > > >, doris::vectorized::MethodStringNoCache > > >&, std::variant, doris::pipeline::ProcessHashTableProbe<2>, doris::pipeline::ProcessHashTableProbe<8>, doris::pipeline::ProcessHashTableProbe<1>, doris::pipeline::ProcessHashTableProbe<4>, doris::pipeline::ProcessHashTableProbe<3>, doris::pipeline::ProcessHashTableProbe<7>, doris::pipeline::ProcessHashTableProbe<9>, doris::pipeline::ProcessHashTableProbe<10>, doris::pipeline::ProcessHashTableProbe<11> >&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013
 6# doris::pipeline::HashJoinProbeOperatorX::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) const at /root/doris/be/src/pipeline/exec/hashjoin_probe_operator.cpp:281
 7# doris::pipeline::StatefulOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/pipeline/exec/operator.cpp:670
 8# doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/pipeline/exec/operator.cpp:381
 9# doris::pipeline::PipelineTask::execute(bool*) in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be
10# doris::pipeline::TaskScheduler::_do_work(int) at /root/doris/be/src/pipeline/task_scheduler.cpp:144
11# doris::ThreadPool::dispatch_thread() at /root/doris/be/src/util/threadpool.cpp:622
12# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:469
13# start_thread at ./nptl/pthread_create.c:442
14# 0x00007F436905A850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```

Related PR: #xxx

Problem Summary:

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-21 12:11:17 +08:00
16b3e5ff18 [improve](information schema) introduce routine load job system table (#48963) (#49286)
pick #48963

Part IV of https://github.com/apache/doris/issues/48511

doc https://github.com/apache/doris-website/pull/2196

**Introduce routine load job statistic system table:**
```
mysql> show create table information_schema.routine_load_job\G
*************************** 1. row ***************************
       Table: routine_load_job
Create Table: CREATE TABLE `routine_load_job` (
  `JOB_ID` text NULL,
  `JOB_NAME` text NULL,
  `CREATE_TIME` text NULL,
  `PAUSE_TIME` text NULL,
  `END_TIME` text NULL,
  `DB_NAME` text NULL,
  `TABLE_NAME` text NULL,
  `STATE` text NULL,
  `CURRENT_TASK_NUM` text NULL,
  `JOB_PROPERTIES` text NULL,
  `DATA_SOURCE_PROPERTIES` text NULL,
  `CUSTOM_PROPERTIES` text NULL,
  `STATISTIC` text NULL,
  `PROGRESS` text NULL,
  `LAG` text NULL,
  `REASON_OF_STATE_CHANGED` text NULL,
  `ERROR_LOG_URLS` text NULL,
  `USER_NAME` text NULL,
  `CURRENT_ABORT_TASK_NUM` int NULL,
  `IS_ABNORMAL_PAUSE` boolean NULL
) ENGINE=SCHEMA;
1 row in set (0.00 sec)
```

**There are some benefits to empower job with SQL query capability for
statistical information:**

- It can be used in conjunction with metrics add through
https://github.com/apache/doris/pull/48209 to roughly locate abnormal
jobs when Grafana alarms, and the following SQL can be used:

```
SELECT JOB_NAME
FROM information_schema.routine_load_job_statistics
WHERE CURRENT_ABORT_TASK_NUM > 0
   OR IS_ABNORMAL_PAUSE = TRUE;
```

- User can use the `select * from information_schema.routine_load_job`
instead of the `show routine load`. The advantage is that the `show
routine load` can only be searched by name, but SQL can be very flexible
in locating jobs

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-21 12:10:34 +08:00
51b39d0992 [fix](join)Consider mark join when computing right_col_idx(#50720) (#50727) 2025-05-19 14:42:15 +08:00
5611a3988b [Fix](JsonPath) return null when meet unknown escape sequence, example '$.name\\k' (#50930)
cherry-pick from #50859
2025-05-17 17:21:22 +08:00
127deb6d2a [Fix](Variant) fix array with predicate push down (#50969)
Cherry-pick from https://github.com/apache/doris/pull/50934
2025-05-17 17:19:47 +08:00
080bc8cbbe branch-2.1: [enhancement](compaction) generate multiple compaction tasks each round #49547 (#50991)
Cherry-picked from #49547

Co-authored-by: Luwei <luwei@selectdb.com>
2025-05-17 17:18:51 +08:00
505c9af95a [fix](inverted index) fix query error (#50860) (#50909)
pick from master #50860
2025-05-17 16:19:15 +08:00
068531129d branch-2.1: [fix](http) remove file before downloading #50754 (#50827)
Cherry-picked from #50754

Co-authored-by: walter <maochuan@selectdb.com>
2025-05-13 15:44:34 +08:00
c5c4b0ac10 branch-2.1: [opt](jdbc scan) Add more jdbc scan profile items #46460 (#50799)
Cherry-picked from #46460

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-05-13 15:43:34 +08:00
0f50cea3d8 branch-2.1: [fix](memory) Fix PODArray::add_num_element (#50785)
pick #50756
2025-05-11 19:14:25 +08:00
1e336ed7ea [fix](restore) avoid change the hard link files of a snapshot (#50753) 2025-05-10 08:27:46 +08:00
0002500757 branch-2.1: [Fix](inverted index) fix rename column build index bug #50056 (#50732)
pick #47562 #50056 from master

---------

Co-authored-by: qiye <luen@selectdb.com>
2025-05-09 17:13:46 +08:00
9422c973af branch-2.1: [fix](binlog) Acquire migration lock before ingesting binlog #50663 (#50709)
cherry pick from #50663
2025-05-09 11:16:18 +08:00
1b108604d5 branch-2.1: [fix](function) fix error result in split_by_string with utf8 chars #40710 (#50689)
Cherry-picked from #40710

Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
2025-05-08 19:15:52 +08:00
5501e130bf [fix](parquet)Fixed the problem that when Parquert reader use index to read files, there will be multiple threads modify same object (#50161) (#50496)
bp #50161
2025-05-08 15:51:25 +08:00
e64e5bc063 [Fix](orc-reader) Fixed issue with top level struct column having present stream failing to access repeatedly when late materialization occurs. (#50651)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note
Cherry-pick #50358

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-07 23:14:11 +08:00
9fe87a56bd [cherry-pick](branch-21) support lead/lag function input column as third params (#49381) (#50653)
### What problem does this PR solve?

cherry-pick from master (#49381)

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-05-07 23:08:56 +08:00
3543124ed3 branch-2.1: [fix](systable) fix 'tables' systable's data_length filling error (#50595)
bp #50593
Also fix a bug that the checking logic for 'TTableStatus' is not right.
This bug only exists in branch-2.1
2025-05-07 23:04:46 +08:00
ebcec779ec branch-2.1: [fix](function) fix error result when input utf8 in url_encode, strright, append_trailing_char_if_absent #49127 (#50660)
…ght, append_trailing_char_if_absent (#49127)

The url_encode function previously performed a modulus operation on a
signed number. Converting it to an unsigned number will fix the issue.
```
before
mysql> select url_encode('编码');
+----------------------+
| url_encode('编码')   |
+----------------------+
| %5.%23%0-%5.%10%/(   |
+----------------------+
now
mysql> select url_encode('编码');
+----------------------+
| url_encode('编码')   |
+----------------------+
| %E7%BC%96%E7%A0%81   |
+----------------------+
```

The strright function did not calculate the length according to the
number of UTF-8 characters.
```
before
mysql> select strright("你好世界",5);
+----------------------------+
| strright("你好世界",5)     |
+----------------------------+
|                            |
+----------------------------+
now

mysql> select strright("你好世界",5);
+----------------------------+
| strright("你好世界",5)     |
+----------------------------+
| 你好世界                   |
+----------------------------+
```

he case of inputting a UTF-8 character was not considered.
```
mysql> select append_trailing_char_if_absent('中文', '文');
+-------------------------------------------------+
| append_trailing_char_if_absent('中文', '文')    |
+-------------------------------------------------+
| NULL                                            |
+-------------------------------------------------+
now
mysql> select append_trailing_char_if_absent('中文', '文');
+-------------------------------------------------+
| append_trailing_char_if_absent('中文', '文')    |
+-------------------------------------------------+
| 中文                                            |
+-------------------------------------------------+
```
2025-05-07 22:37:50 +08:00
02c3157e4c [branch-2.1](function) fix wrong floor of function date_diff when unit less than day (#49429) (#50606)
pick https://github.com/apache/doris/pull/49429
2025-05-07 09:27:37 +08:00
babb98f10c [branch-2.1](function) fix wrong length check of function array_range (#49993) (#50605)
pick https://github.com/apache/doris/pull/49993
2025-05-06 18:57:03 +08:00
d667b5af38 [branch-2.1](function) Fix unexpected result of unix_timestamp when input is out of bound (#49430) (#50609)
pick https://github.com/apache/doris/pull/49430 but removed be-ut
because we can't.

Co-authored-by: Pxl <xl@selectdb.com>
2025-05-06 17:10:55 +08:00
65151b46aa branch-2.1: [improve](routine load) add more metrics to observe the routine load job #48209 (#48765)
Cherry-picked from #48209

Co-authored-by: hui lai <laihui@selectdb.com>
2025-05-05 10:25:37 +08:00
Pxl
6dee1fe623 [Chore](column) change ColumnFixedLengthObject log fatal to throw exception (#50540) 2025-05-03 23:04:01 +08:00
af4195e399 branch-2.1: [fix](geo) Fix ST_Contains behavior #50115 (#50569)
Cherry-picked from #50115

Co-authored-by: linrrarity <142187136+linrrzqqq@users.noreply.github.com>
2025-05-03 22:36:22 +08:00
4b3dd6c10a branch-2.1: [feat](func) any function supports json #50311 (#50484)
Cherry-picked from #50311

Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com>
2025-04-29 19:11:25 +08:00
5be2556503 [enhancement](compaction) replace hardcoded compaction thresholds with config (#49662) (#50514)
pick master #49662
2025-04-29 16:47:18 +08:00
3660139c64 branch-2.1: [fix](path gc) Fix path gc race with publish task #50343 (#50520)
cherry pick from #50343
2025-04-29 16:17:34 +08:00
4e8148105a [fix](serde)Fixed the issue that serde may cause be core when reading schema changed text table. (#50105) (#50504)
bp #50105
2025-04-28 21:54:43 -07:00
8cd9fd1e89 branch-2.1: [fix](memory) Fix thread_mem_tracker() performance (#50524) 2025-04-29 10:22:32 +08:00
b3e55d1064 [log](mow) reduce log for data load on mow table (#47647) (#50491)
cherry-pick #47647
2025-04-28 19:07:19 +08:00
619a491e98 branch-2.1: [Fix](mow) Fix DeleteBitmap::cardinality (#50404)
### What problem does this PR solve?


```
*** Query id: ce47f2673e782ef9-5581cc671c6e7c85 ***
*** is nereids: 0 ***
*** tablet id: 10196 ***
*** Aborted at 1745534277 (unix time) try "date -d @1745534277" if you are using GNU date ***
*** Current BE git commitID: e5a9c2552d ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 5684 (TID 117604 OR 0x7fe01cffd640) from PID 0; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/common/signal_handler.h:421
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007FF412552520 in /lib/x86_64-linux-gnu/libc.so.6
 4# memcpy at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
 5# array_container_clone in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
 6# ra_overwrite in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
 7# roaring::Roaring::Roaring(roaring::Roaring const&) at /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/roaring/roaring.hh:99
 8# doris::DeleteBitmap::cardinality() const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/tablet_meta.cpp:1072
 9# doris::BetaRowsetWriter::_generate_delete_bitmap(int) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/rowset/beta_rowset_writer.cpp:198
10# doris::BaseBetaRowsetWriter::add_segment(unsigned int, doris::SegmentStatistics const&, std::shared_ptr<doris::TabletSchema>) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/rowset/beta_rowset_writer.cpp:883
11# doris::BetaRowsetWriter::add_segment(unsigned int, doris::SegmentStatistics const&, std::shared_ptr<doris::TabletSchema>) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
12# doris::SegmentCollectorT<doris::BaseBetaRowsetWriter>::add(unsigned int, doris::SegmentStatistics&, std::shared_ptr<doris::TabletSchema>) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/rowset/segment_creator.h:82
13# doris::SegmentFlusher::_flush_segment_writer(std::unique_ptr<doris::segment_v2::VerticalSegmentWriter, std::default_delete<doris::segment_v2::VerticalSegmentWriter> >&, std::shared_ptr<doris::TabletSchema>, long*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/rowset/segment_creator.cpp:247
14# doris::SegmentFlusher::flush_single_block(doris::vectorized::Block const*, int, long*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/rowset/segment_creator.cpp:78
15# doris::SegmentCreator::flush_single_block(doris::vectorized::Block const*, int, long*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/rowset/segment_creator.cpp:400
16# doris::BaseBetaRowsetWriter::flush_memtable(doris::vectorized::Block*, int, long*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/rowset/beta_rowset_writer.cpp:539
17# doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/memtable_flush_executor.cpp:158
18# doris::FlushToken::_flush_memtable(std::unique_ptr<doris::MemTable, std::default_delete<doris::MemTable> >, int, long) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
19# doris::MemtableFlushTask::run() at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/memtable_flush_executor.cpp:63
20# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
21# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:499
22# start_thread at ./nptl/pthread_create.c:442
23# 0x00007FF412636850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83

```
2025-04-26 14:57:38 +08:00
0710d9b2d6 branch-2.1: [fix](orc) Should not pass selection vector when decode child column of List or Map #50136 (#50316)
bp: #50136
2025-04-25 09:04:06 +08:00
90e22197ff [fix](json-functions)fix json-replace/insert/set/array behavior with complex type #50308 (#50309)
### What problem does this PR solve?
backport: https://github.com/apache/doris/pull/50308

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-04-24 19:04:25 +08:00
bc68a8da07 branch-2.1: remove visible rowset from memory during deletion transaction (#50329) 2025-04-24 09:41:37 +08:00
cf72fa82e2 [Improve](explode) explode function support multi param (#50310)
### What problem does this PR solve?
backport:https://github.com/apache/doris/pull/48537
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-04-23 23:27:07 +08:00