Commit Graph

20790 Commits

Author SHA1 Message Date
ec068049c7 branch-2.1: [fix](meta-cache) fix refreshOnlyCatalogCache when use_meta_cache = false #44363 (#44461)
Cherry-picked from #44363

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-11-22 22:41:10 +08:00
6af6d4a5ef branch-2.1: [refactor](regression) Re-arrange cases for data-partitioning doc (#44248)
pick https://github.com/apache/doris/pull/43373
2024-11-22 19:49:07 +08:00
7d123edcf8 [fix](filesystem)Use simple authentication directly in S3FileSystem for 2.1 (#43636) (#44238)
bp: #43636
2024-11-22 11:45:56 +08:00
cd223327c8 [bug](auto partition)Fix insert failed because partition exprs of table is empty (#44391)
Fix insert failed because partition exprs of table is empty.

Revert PR: #43274
2024-11-22 11:16:23 +08:00
702abbff0f [Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. (#42004) (#44239)
bp #42004

Co-authored-by: kaka11chen <kaka11.chen@gmail.com>
2024-11-22 11:01:41 +08:00
75f8323f45 [opt](regression-test) move some cases to p2 to accelerate the testing speed of p0 (#37590) (#44389)
pick #37590

move some cases to p2 to accelerate the testing speed of p0
2024-11-21 23:07:57 +08:00
88c258c218 [test](routine load) fix some unstable routine load cases (#42252) (#44388)
pick #42252

Fix some unstable routine load case.
2024-11-21 23:07:01 +08:00
4ae4833bdc [test](routine load) add routine load doc case (#43230) (#44382)
pick #43230

Add routine load doc case.
2024-11-21 23:06:14 +08:00
346b89e683 [improve](routine load) adjust default values to make routine load more convenient to use (#42491) (#44377)
pick (#42491)

For a routine load job, it will be divided into many tasks, each of
which is a transaction. Currently, the default time
consumed(max_batch_interval) is 10 seconds. The benefits of increasing
this value are:
1. Larger batch consumption can lead to better performance.
2. Reducing the number of transactions can alleviate the pressure of
compaction and the conflicts of concurrent transaction submissions.

related doc: https://github.com/apache/doris-website/pull/1236/files
2024-11-21 23:05:11 +08:00
a258915e22 branch-2.1: [Fix](Job)Fix the Calculation of the First Trigger Time and Add a Single-Time Scheduling Compensation Logic #44268 (#44403)
Cherry-picked from #44268

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
2024-11-21 18:24:36 +08:00
eb975f7846 [improvement](statistics)Change auto analyze max width to 300 and health threshold to 90. (#42104) (#44379)
backport: https://github.com/apache/doris/pull/42104
2024-11-21 17:35:41 +08:00
936417e6f9 [cherry-pick][test](doc) add table design example in doris's doc to regression test (#42444) (#44313)
(cherry picked from commit 051128f172a0e6c26109160752293d95746e29f8)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #42444

Problem Summary:
2024-11-21 14:58:42 +08:00
fbf573f8c8 [fix](hive)fix hive catalog miss partition that have special characters. #42906 (#44152)
cherry pick from #42906

---------

Co-authored-by: daidai <2017501503@qq.com>
2024-11-21 14:11:22 +08:00
9664b50eb6 [improve](load) do not block delta writer if memtable memory is low (#42649) (#44305)
backport #42649
2024-11-21 11:17:35 +08:00
3c1ac52f6a [fix](nereids)should set isForwardedToMaster and redirectStatus to null before fallback to legacy planner (#44332)
pick from master https://github.com/apache/doris/pull/44163
2024-11-21 10:25:35 +08:00
fb163b55c2 branch-2.1: [Fix](merge-on-write) Fix MergeIndexDeleteBitmapCalculator::calculate_one() coredump #44284 (#44330)
Cherry-picked from #44284

Co-authored-by: bobhan1 <baohan@selectdb.com>
2024-11-20 21:07:43 +08:00
b63b97af25 [fix](create table) Enhance the robustness of time zone handling during table creation #41926 (#44281)
cherry pick from #41926
2024-11-20 10:37:54 +08:00
dc67086d97 [fix](scan) Avoid memory allocated by buffered_reader from being traced (#41921) (#44253)
Use OwnedSlice to replace `char*` in BufferedReader

## Proposed changes

pick #41921
2024-11-20 10:37:06 +08:00
a666b5e0c9 [fix](case) Enhance the robustness of set_config_temporary_action #42306 (#44282)
cherry pick from #42306
2024-11-20 10:35:44 +08:00
e0261ffb9c [fix](auth)Fix all priv not contain Show_view_priv (#43605) (#44289)
pick: https://github.com/apache/doris/pull/43605
2024-11-20 10:33:49 +08:00
0ddf2958b3 [fix](trash) Fix clean_trash docker regression case #40925 (#44280)
cherry pick from #40925
2024-11-20 09:41:45 +08:00
1913e4e78d branch-2.1: [fix](mtmv) Fix partition track column fail when 'select *' used in related partition side #43531 (#43941)
Cherry-picked from #43531

Co-authored-by: seawinde <wusi@selectdb.com>
2024-11-19 20:27:13 +08:00
3136fa48a6 branch-2.1: [chore](ci) adjust some invalid url #44261 (#44270)
Cherry-picked from #44261

Co-authored-by: Dongyang Li <lidongyang@selectdb.com>
2024-11-19 19:28:04 +08:00
1b1e2e8a29 branch-2.1: [fix](profile) only printed for non-sink nodes in the merge profile. #44040 (#44252)
Cherry-picked from #44040

Co-authored-by: Mryange <yanxuecheng@selectdb.com>
2024-11-19 17:34:38 +08:00
2eb66c2609 [Fix](PaimonCatalog) fix the problem that paimon catalog can not access to OSS-HDFS #42585 (#44149)
cherry pick from #42585

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
2024-11-19 17:31:42 +08:00
610054c77b [cherry-pick](branch-21) fix exchange of tablet shuffle send block error (#44102) (#44230)
cherry-pick from master (#44102)
2024-11-19 17:31:06 +08:00
fb20b7366e [fix](roup commit) remove unused code (#44015) (#44118)
pick https://github.com/apache/doris/pull/44015
2024-11-19 17:29:59 +08:00
81a8d9cb25 [fix](nereids)the column name should be case insensitive in tablet prune (#44206) 2024-11-19 17:28:46 +08:00
433c1bc9ff [cherry-pick](branch-21) replace the LOG(FATAL) to throw Exception in query execute layer (#38144) (#44183)
cherry-pick from master https://github.com/apache/doris/pull/38144
2024-11-19 17:28:20 +08:00
c9801f7a38 branch-2.1: [Bug](function) fix cut_ipv6 function error about modify the input column data #43921 (#44180)
Cherry-picked from #43921

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
2024-11-19 17:27:29 +08:00
83208ee1a8 [pick](branch-2.1) pick #43960 #43929 #44177 (#44240)
pick #43960 #43929 #44177
2024-11-19 17:25:16 +08:00
83b74827aa branch-2.1: [fix](iceberg)Fix count(*) error with dangling delete problem #44039 (#44101)
Cherry-picked from #44039

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
2024-11-19 17:19:25 +08:00
e9371325bd branch-2.1: [fix](s3) do not replace https scheme if specified #44242 (#44250)
Cherry-picked from #44242

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2024-11-19 17:12:42 +08:00
eeafe45c0c [fix](brpc) coredump caused by brpc checking (#44047) (#44188)
pick #44047
```
/root/doris/be/src/runtime/fragment_mgr.cpp:1064:20: runtime error: member call on null pointer of type 'doris::PBackendService_Stub'

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/runtime/fragment_mgr.cpp:1064:20 in
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1731663847 (unix time) try "date -d @1731663847" if you are using GNU date ***
*** Current BE git commitID: b663df0e50 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 17169 (TID 17463 OR 0x7f746d21a700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo_t*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
3# 0x00007F7601263090 in /lib/x86_64-linux-gnu/libc.so.6
4# doris::FragmentMgr::_check_brpc_available(std::shared_ptr<doris::PBackendService_Stub> const&, doris::FragmentMgr::BrpcItem const&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
5# doris::FragmentMgr::cancel_worker() at /root/doris/be/src/runtime/fragment_mgr.cpp:1022
6# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:499
7# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
8# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2024-11-19 14:56:46 +08:00
b4e136bfcc [performance](move-memtable) async close tablet streams (#41156 & #43813) (#44128)
backport #41156 and #43813
2024-11-19 14:14:53 +08:00
9db3f5b695 [refactor](hudi, iceberg) optimize some code #42636 (#44148)
cherry pick from #42636
2024-11-19 13:55:02 +08:00
ab7f5ba2ac branch-2.1: [fix](mtmv)After failure, one should quickly return to avoid repeated failures #44174 (#44204)
Cherry-picked from #44174

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2024-11-19 11:05:15 +08:00
2cbfa02a5f [chore](autopick)Delete Auto Pick Branch (#44225) 2024-11-19 10:44:19 +08:00
c3dc1863bd [opt](binlog) Support drop view binlog #39781 (#44113)
cherry pick from #39781

Co-authored-by: smallx <e9999e@163.com>
2024-11-19 10:41:34 +08:00
cb40a0bee3 branch-2.1: [test](statistics)Add stats test for replace table. #44011 (#44212)
Cherry-picked from #44011

Co-authored-by: James <lijibing@selectdb.com>
2024-11-19 07:57:13 +08:00
efb3bdd96e [fix](test) fix clickhouse jdbc catalog func push down case #43196 (#44151)
cherry pick from #43196

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-11-18 18:03:10 +08:00
161d76328d branch-2.1: [fix](iceberg)Delete the data files when deleting the table #43441 (#44138)
Cherry-picked from #43441

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
2024-11-18 18:00:13 +08:00
b01b202b93 branch-2.1: [fix](Nereids) fold const return type does not matched with type coercion #44022 (#44141)
Cherry-picked from #44022

Co-authored-by: LiBinfeng <libinfeng@selectdb.com>
2024-11-18 17:57:15 +08:00
05e162074c branch-2.1: [fix](Export) fix a memory leak in the FE because of the ExportJob #44019 (#44089)
Cherry-picked from #44019

Co-authored-by: Tiewei Fang <ftw2139@163.com>
2024-11-18 16:54:53 +08:00
abbb12f93f branch-2.1: [fix](Nereids) simplify comparison predicate do wrong cast #44054 (#44119)
Cherry-picked from #44054

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2024-11-18 16:53:51 +08:00
7f129433ec [improve](binlog) Reserve ids for binlog type allocation #43769 (#44103)
cherry pick from #43769
2024-11-18 16:17:19 +08:00
b776c2a983 branch-2.1: [fix](binlog) add modify comment binlog when replaying journal #43827 (#43986)
Cherry-picked from #43827

Co-authored-by: walter <maochuan@selectdb.com>
2024-11-18 15:06:25 +08:00
fa43cc0a90 [improve](http) Save the requested url in http execution error #43855 (#44106)
cherry pick from #43855
2024-11-18 15:06:02 +08:00
6c796af81b [improve](backup) Add the commit seq at the backup job level. #44049 (#44110)
cherry pick from #44049
2024-11-18 15:00:08 +08:00
8da1e8c084 branch-2.1: [feat](catalog)Support Pre-Execution Authentication for HMS Type Iceberg Catalog Operations. #43445 (#44129)
Cherry-picked from #43445

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
2024-11-18 14:28:25 +08:00