Commit Graph

21155 Commits

Author SHA1 Message Date
7e615afbce branch-2.1: [fix](regression-test) fix test_outfile_csv_with_names_and_types case #46214 (#46272)
Cherry-picked from #46214

Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
2025-01-02 15:06:39 +08:00
b5beaf4eb1 branch-2.1: [fix](mtmv)The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated #46020 (#46265)
Cherry-picked from #46020

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-01-02 14:03:28 +08:00
ffb8077ecb [Chore](regression-test) adjust variant tpch/q09_trans.sql batch_size… (#46249)
… from default to 2048 (#34105)

since 50 may cause performance issue introduced by #33853

Co-authored-by: lihangyu <15605149486@163.com>
2025-01-02 10:50:12 +08:00
5deeb42f6d branch-2.1: [opt] Optimization for short circuit of CompoundPred #45422 (#46241)
cherry pick from #45422
2025-01-02 10:11:47 +08:00
8e730faec5 [Exec](expr) Opt the compound pred performace (#45414) (#46232)
cherry-pick #45414

before:
```
 mysqlslap -hd3 -uroot -P9130  --create-schema=test_db2 -c 10 -i 500 -q "SELECT count(k) FROM sbtest1_dup WHERE k BETWEEN 4850578 AND 8454295 OR k BETWEEN 8776291 AND 29749077;"
Benchmark
	Average number of seconds to run all queries: 0.041 seconds
	Minimum number of seconds to run all queries: 0.037 seconds
	Maximum number of seconds to run all queries: 0.115 seconds
	Number of clients running queries: 10
	Average number of queries per client: 1
```

after:
```
mysqlslap -hd3 -uroot -P9030  --create-schema=test_db -c 10 -i 500 -q "SELECT count(k) FROM sbtest1 WHERE k BETWEEN 4850578 AND 8454295 OR k BETWEEN 8776291 AND 29749077;"
Benchmark
	Average number of seconds to run all queries: 0.029 seconds
	Minimum number of seconds to run all queries: 0.027 seconds
	Maximum number of seconds to run all queries: 0.034 seconds
	Number of clients running queries: 10
	Average number of queries per client: 1
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-01-01 16:19:42 +08:00
726e1c8c80 [opt](profile) add summary metric for file scanner #45941 (#46188)
bp #45941
2025-01-01 13:27:50 +08:00
9e45c70923 branch-2.1: [chore](restore) log the table name and partition info when partition type is different #46213 (#46234)
Cherry-picked from #46213

Co-authored-by: walter <maochuan@selectdb.com>
2025-01-01 08:09:11 +08:00
a6fc852ce7 branch-2.1: [opt](catalog) use table in db object return get db #46211 (#46229)
Cherry-picked from #46211

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-authored-by: morningman <yunyou@selectdb.com>
2025-01-01 08:00:49 +08:00
0ee1a6273f [branch-2.1](regression) disable hudi p2 jni reader test for branch-2.1 (#46231) 2025-01-01 08:00:15 +08:00
9593a7bfc3 branch-2.1: [fix](split)Fixed the bug that batch mode split could not query data in multiple be scenarios. #46218 (#46227)
Cherry-picked from #46218

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-01-01 00:34:48 +08:00
2f6c13d4c0 branch-2.1: [regression-test](fix) remove interrupt() in test_routine_load_eof.groovy #46044 (#46098)
Cherry-picked from #46044

Co-authored-by: shuke <shuke@selectdb.com>
2024-12-31 23:17:46 +08:00
84126ea721 [regression-test](fix) fix test_cumu_compaction_with_delete.groovy global var (#46165) 2024-12-31 23:08:04 +08:00
e4adf9b931 [fix](routine load) replace heavy work pool with routine load thread pool for metadata fetching (#44907) (#46186)
pick #44907

In production, we encountered an issue where the librdkafka consumer
stucked during destruction, causing the heavy work pool to become
saturated, which in turn made all heavy work pool-dependent
functionalities, such as querying, unusable. To mitigate this impact, we
replaced the heavy work pool with routine load threads for metadata
fetching.
2024-12-31 23:06:08 +08:00
28960b7b7d [Improve](nereids) use hash set replace three set in DiscreteValue to improve in predicate performance (#45181) (#46220)
### What problem does this PR solve?

cp #45181
2024-12-31 23:04:57 +08:00
1d8b3a0bcc branch-2.1: [fix](audit) fix the delay in loading audit log #46175 (#46193)
Cherry-picked from #46175

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
2024-12-31 22:56:12 +08:00
f59c285538 branch-2.1: [fix](index)Add duplicated indexes check in add index #46155 (#46210)
Cherry-picked from #46155

Co-authored-by: qiye <luen@selectdb.com>
2024-12-31 22:46:29 +08:00
fbfb36b8e2 branch-2.1: [test](catalog) add lower case meta auth test #46114 (#46129)
Cherry-picked from #46114

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-12-31 22:10:11 +08:00
9390c743c5 branch-2.1: [Fix](multi-catalog) Fix column mutate() crash replace it by assume_mutable(). #46151 (#46198)
Cherry-picked from #46151

Co-authored-by: Qi Chen <chenqi@selectdb.com>
2024-12-31 22:02:49 +08:00
9eb07e578e branch-2.1 [Fix](catalog)Remove the fs.disable.cache parameter to prevent excessive FS-associated objects and memory leaks (#46184) (#46191)
#46184
(cherry picked from commit 17667aeb92efc5580b8615e53a3ef4f514a27204)
2024-12-31 16:54:10 +08:00
fa20529dd9 [enhance](auth)Optimize the authentication logic of Ranger Doris(#41207) (#46179)
pick: https://github.com/apache/doris/pull/41207

only pick Performance optimization related
2024-12-31 11:54:27 +08:00
02239e4fb2 branch-2.1: [chore](regression) do not hard code S3 bucket and endpoint of hive t… #46159 (#46169)
Cherry-picked from #46159

Co-authored-by: zgxme <zhenggaoxiong@selectdb.com>
2024-12-31 11:44:36 +08:00
4472648c07 [branch-2.1] pick workload group usage metrics (#46177)
pick #45284  #44870
2024-12-31 10:09:48 +08:00
aa8c93cfe2 branch-2.1: [fix](sql) Wrong result for alter set property #46028 (#46141)
pick https://github.com/apache/doris/pull/46028
2024-12-31 09:52:43 +08:00
df26475e1a [Enhancement](compaction) enable the compaction producer to generate multiple compaction tasks in a single run (#45411) (#46160)
pick master #45411
2024-12-31 09:51:43 +08:00
3d79955db3 branch-2.1: [fix](parquet-reader) Fixed the issue of excessive scanning data in late materialization‌ case of parquet reader #46121 (#46183)
Cherry-picked from #46121

Co-authored-by: Qi Chen <chenqi@selectdb.com>
2024-12-31 07:30:49 +08:00
1af18d0211 [fix](audit-loader) Fix the bug that causes audit loader to fail. (#46119)
Fix the bug that causes audit loader to fail.
Related PR: #45167 #40493

The bug causes audit loader fail as following errors in audit.log.
```
2024-12-27 11:47:47,001 [stream_load] |Label=audit_log_20241227_114552_856_127_0_0_1_8030|Db=__internal_schema|Table=audit_log|User=|ClientIp=10.0.1.3|Status=Success|Message=OK|Url=http://10.0.1.4:8040/api/_load_error_log?file=__shard_7/error_log_insert_stmt_c24ed0d941f59867-ec08b8542bc2a4a1_c24ed0d941f59867_ec08b8542bc2a4a1|TotalRows=34|LoadedRows=0|FilteredRows=34|UnselectedRows=0|LoadBytes=6887|StartTime=2024-12-27 11:45:52.858|FinishTime=2024-12-27 11:45:52.888
```
The detail error is:
```
curl http://10.0.1.4:8040/api/_load_error_log?file=__shard_7/error_log_insert_stmt_c24ed0d941f59867-ec08b8542bc2a4a1_c24ed0d941f59867_ec08b8542bc2a4a1

Reason: actual column number in csv file is  more than  schema column number.actual number: 29, schema column number: 27; line delimiter: [
], column separator: [  ], result values:
```

Co-authored-by: derenli <derenli@tencent.com>
2024-12-31 07:30:00 +08:00
Pxl
43c646363e [Bug](runtime-filter) support ip rf and use exception to replace dche… (#41531)
…ck when PrimitiveType to PColumnType (#39985)

use exception to replace dcheck when PrimitiveType to PColumnType
```cpp
*** SIGABRT unknown detail explain (@0x11d3f) received by PID 73023 (TID 74292 OR 0x7fd758225640) from PID 73023; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
 1# 0x00007FDDBE6B9520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill at ./nptl/pthread_kill.c:89
 3# raise at ../sysdeps/posix/raise.c:27
 4# abort at ./stdlib/abort.c:81
 5# 0x000056123F81A94D in /root/output/be/lib/doris_be
 6# 0x000056123F80CF8A in /root/output/be/lib/doris_be
 7# google::LogMessage::SendToLog() in /root/output/be/lib/doris_be
 8# google::LogMessage::Flush() in /root/output/be/lib/doris_be
 9# google::LogMessageFatal::~LogMessageFatal() in /root/output/be/lib/doris_be
10# doris::to_proto(doris::PrimitiveType) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:114
11# doris::IRuntimeFilter::push_to_remote(doris::TNetworkAddress const*) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:1143
12# doris::IRuntimeFilter::publish(bool)::$_0::operator()(doris::IRuntimeFilter*) const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:959
13# doris::IRuntimeFilter::publish(bool)::$_2::operator()() const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:983
14# doris::IRuntimeFilter::publish(bool) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:997
```

## Proposed changes
pick from #39985
2024-12-30 20:56:11 +08:00
c6742cbe7e branch-2.1: [fix](test) Inverted index property inconsistent #45834 (#46143)
pick: https://github.com/apache/doris/pull/45834
2024-12-30 20:05:13 +08:00
419456f3a9 branch-2.1: [fix](sort)fix merge sort may miss the limit #46072 (#46158)
Cherry-picked from #46072

Co-authored-by: Mryange <yanxuecheng@selectdb.com>
2024-12-30 20:02:24 +08:00
10ad255198 branch-2.1: [fix](memory) Fix purge jemalloc dirty page (#46146)
### What problem does this PR solve?

Fix purge jemalloc dirty page
2024-12-30 20:00:54 +08:00
d5b145ab07 [regression-test](fix) rm unused case regression-test/suites/nereids_rules_p0/defer_materialize_topn/one_phase.groovy (#46167) 2024-12-30 17:37:48 +08:00
7040abfb04 [fix](correctness) Fix operator initialization (#45728) (#46150)
Now we plan for local exchange depends on operator initialization. This
PR fixed a wrong order between those two steps.

pick #45728
2024-12-30 17:01:33 +08:00
c8fb934bc1 branch-2.1: [chore](config) disable restore_reset_index_id by default #46104 (#46127)
Cherry-picked from #46104

Co-authored-by: walter <maochuan@selectdb.com>
2024-12-30 11:52:58 +08:00
6c9167d9d9 branch-2.1: [test](doc) add job-scheduler example in doris's doc to regression test #42904 (#45968)
Cherry-picked from #42904

Co-authored-by: yagagagaga <zhangminke@selectdb.com>
2024-12-30 10:44:58 +08:00
5425e4686b branch-2.1: [fix](auth) fix use database stmt access unauthorized catalog #45720 (#45978)
Cherry-picked from #45720

Co-authored-by: Petrichor <xiaowenli@selectdb.com>
2024-12-30 10:43:28 +08:00
a835203947 branch-2.1: [regression-test](fix) enlarge statistic test case's waiting time #45590 (#46017)
Cherry-picked from #45590

Co-authored-by: shuke <shuke@selectdb.com>
2024-12-30 10:41:55 +08:00
9b91a4a560 branch-2.1: [test](nereids) add test for normalize sort #45890 (#46112)
Cherry-picked from #45890

Co-authored-by: yujun <yujun@selectdb.com>
2024-12-30 10:19:40 +08:00
1d742b5f7d [Cherry-pick](branch-2.1) Pick "[Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset (#43466)" (#46117)
Before this PR, in cases where there is an alternating distribution of
data rowset -> delete rowset -> data rowset -> delete rowset, cumulative
compaction would only move the cumulative point forward to allow base
compaction to handle the delete rowset. Cumulative compaction itself
would not process the data and would return be marked as failure. This
would cause the compaction submission task process to pause for 5
seconds, impacting efficiency.

This PR modifies the return status to OK for such cases, which improves
the efficiency of the compaction submission task.
2024-12-30 10:18:57 +08:00
b085e94a5e branch-2.1: [fix](sql) Wrong result for partition item tosql #45918 (#45960)
Cherry-picked from #45918

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2024-12-30 10:09:58 +08:00
a490a366e4 [improvement](statistics)Support get oracle jdbc row count. (#45214) (#46133)
backport: https://github.com/apache/doris/pull/45214
2024-12-29 22:17:57 +08:00
6dd92be33d [feature](statistics)Support get row count for pg and sql server. (#42674) (#46131)
backport: https://github.com/apache/doris/pull/42674
2024-12-29 19:37:21 +08:00
a380f5d222 [enchement](utf8)import enable_text_validate_utf8 session var (#45537) (#46070)
bp #45537
2024-12-28 10:05:03 +08:00
829d9b3454 branch-2.1: [Fix](PreparedStatement) nondeterministic functions in prepared statement should not be short circuited #46003 (#46109)
Cherry-picked from #46003

Co-authored-by: lihangyu <lihangyu@selectdb.com>
2024-12-27 21:02:36 +08:00
75e9aa3fb4 branch-2.1: [fix](jdbc catalog) Get statistics using remote name #46040 (#46116)
Cherry-picked from https://github.com/apache/doris/pull/46040
2024-12-27 20:59:18 +08:00
8ac54b63c5 branch-2.1 [Fix](Catalog)Standardize the use of authentication attributes in the Catalog to avoid creating a large number of Authenticator objects. (#46106)
…utes in the Catalog to avoid creating a large number of Authenticator
objects.

#46052
2024-12-27 20:36:12 +08:00
fab89c199e branch-2.1: [chore](binlog) Add table type to getMeta method #45942 (#45972)
Cherry-picked from #45942

Co-authored-by: walter <maochuan@selectdb.com>
2024-12-27 20:03:53 +08:00
a1840a0630 [cherry-pick](branch-2.1)regression-test alter table partition (#43561)
prick from :https://github.com/apache/doris/pull/41437
2024-12-27 19:48:07 +08:00
6cee95540d branch-2.1: [fix](variant) disable column name with dot character for variant type #45927 (#45992)
Cherry-picked from #45927

Co-authored-by: camby <cambyzhu@tencent.com>
2024-12-27 19:12:44 +08:00
4746e9e3a2 [opt](inverted index)Optimize code to get rid of heap use after free (#45745) (#46075)
bp #45745
2024-12-27 16:46:58 +08:00
69961a3c1b [opt](hms table)Some optimizations for hms external table for 2.1 (#44909) (#46084)
bp: #44909
2024-12-27 16:42:42 +08:00