Commit Graph

19689 Commits

Author SHA1 Message Date
843afccdf9 [fix](catalog) remove backend in black list from candidate backends for external table (#38984)
When select backends for external table's query,
skip the backends in black list
2024-08-07 14:09:06 +08:00
f9788b4ca5 [Fix](nereids) fix partition_prune or expression evaluate wrongly (#38897) (#38998)
cherry-pick #38897 to branch-2.1
2024-08-07 13:49:42 +08:00
7550fbaff7 [Fix](Exception) throw exception in defer may result std::terminate (… (#39007)
pick #38935
2024-08-07 13:46:23 +08:00
8cb5aa64f4 [test](inverted index) add an Inverted Index Testing Switch (#38077) (#38947)
https://github.com/apache/doris/pull/38077
2024-08-07 11:25:36 +08:00
fc0222a64c [opt](info) processlist schema table support show all fe (#38701) (#38953)
pick #38701
2024-08-07 11:01:46 +08:00
b856530b09 [fix](inverted index) disable range query in StringTypeInvertedIndexReader (#38218) (#38926)
## Proposed changes

pick from master #38218

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-07 10:44:02 +08:00
e400859531 [fix](update null map) Fix update_null_map #38787 (#38920)
cherry pick from #38787
2024-08-07 10:21:41 +08:00
2b1aa05370 pick some pr from to branch21 #38115 #38008 #37929 (#38940)
## Proposed changes

pr: https://github.com/apache/doris/pull/38115
commitId: 2b29288c

pr: https://github.com/apache/doris/pull/38008
commitId: c6b924da

pr: https://github.com/apache/doris/pull/37929
commitId: d44fcdc5
2024-08-07 10:19:41 +08:00
2543b569bb [Optimize](Row store) pick #37145, #38236 (#38932) 2024-08-07 09:55:42 +08:00
bc644cb253 [opt](catalog) merge scan range to avoid too many splits (#38311) (#38964)
bp #38311
2024-08-06 21:57:02 +08:00
2540835b58 [opt](log) Remove unnecessary log for analysis (#38943)
In master branch, we already fixed in this pr:
https://github.com/apache/doris/pull/36884

Here cherry-pick to branch-2.1
2024-08-06 21:44:18 +08:00
07ea511141 [opt](optimizer) Remove unused code to unify code (#38918)
## Proposed changes
Now, Agg's child predicates will  not spread to agg.
    For example:
    select a, sum(b)
    from (
     select a,b from t where a = 1 and b = 2
    ) t
    group by a
    `a = 1` in scan can be propagated to `a` of agg.
    But `b = 2` in scan can not be propagated to `sum(b)` of agg.

Issue Number: #38905
<!--Describe your changes.-->

Co-authored-by: liutang123 <liulijia@gmail.com>
2024-08-06 19:09:25 +08:00
3abb222064 [fix](group commit) Fix test_group_commit_async_wal_msg_fault_injection case (#35313) (#38911)
pick https://github.com/apache/doris/pull/35313
2024-08-06 17:57:22 +08:00
fe6ea3b8b5 [Fix](inverted index) fix missed array inverted index null bitmap #38907 (#38934)
cherry pick from #38907
2024-08-06 17:17:28 +08:00
21a67dba5d [fix](index) fix inverted index compound file entry size int32 overflow #38891 (#38928) 2024-08-06 15:57:09 +08:00
5066be6df3 [fix](multicatalog) fix hadoop authenticator not inited for existing hms catalog. (#38930)
Backport #38475.

Co-authored-by: Yulei-Yang <yulei.yang0699@gmail.com>
2024-08-06 15:34:32 +08:00
28c0510440 [fix](pipeline) Fix mem control in local exchanger (#38885) (#38910)
If a block (>128M) is dequeue by local exchange source operator and it
is the last block, both of source operators and sink operators will be
hang. This PR fixed it.

pick #38885
2024-08-06 14:45:41 +08:00
ba5c6fba98 [scheduler](core) Use signed int as number of cores (#38514) (#38913)
pick #38514

*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1722279016 (unix time) try "date -d @1722279016" if you
are using GNU date ***
*** Current BE git commitID: e9f12fac47e ***
*** SIGSEGV unknown detail explain (@0x0) received by PID 1116227 (TID
1116498 OR 0x7f009ac00640) from PID 0; stack trace: *** 0#
doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at

/home/zcp/repo_center/doris_branch-2.1/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2#
JVM_handle_linux_signal in
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F01E49B0520 in /lib/x86_64-linux-gnu/libc.so.6
 4# pthread_mutex_lock at ./nptl/pthread_mutex_lock.c:80
5# doris::pipeline::MultiCoreTaskQueue::take(unsigned long) at
/home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:154
6# doris::pipeline::TaskScheduler::_do_work(unsigned long) at
/home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:268
7# doris::ThreadPool::dispatch_thread() in
/mnt/disk1/STRESS_ENV/be/lib/doris_be
8# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:499
 9# start_thread at ./nptl/pthread_create.c:442
10# 0x00007F01E4A94850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-06 14:44:59 +08:00
8ce30963cd [fix] (compaction) fix time series compaction policy (#38220) (#38917)
## Proposed changes

pick from #38220
2024-08-06 14:26:42 +08:00
fcb4483ed1 [chore](sql) Forbid show hidden columns and create table with hidden column (#38796) (#38924)
Forbid show hidden columns and create table with hidden column
2024-08-06 14:24:41 +08:00
75fe929dc4 [enhancement](nereids) Support eliminate outer join by match expression (#38537) (#38925)
enable run match expression outer of filter plan, e.g join conjunct
support eliminate outer join by match expression, if any arguments of match expression is null literal
2024-08-06 13:16:57 +08:00
ff6fa33021 [opt](inverted index) mow supports index optimization #(#38180)
## Proposed changes

https://github.com/apache/doris/pull/37428
https://github.com/apache/doris/pull/37429

<!--Describe your changes.-->
2024-08-06 11:18:13 +08:00
ab3057b2d4 [Feat](nereids) support date function in partition prune (#38743) (#38898)
cherry-pick #38743 to branch-2.1
2024-08-06 09:13:13 +08:00
bcea54147c [feature](inverted index) String type inverted index match function c… (#38872)
https://github.com/apache/doris/pull/38170
2024-08-06 09:06:05 +08:00
c7b59b38ef [fix](hist) Fix unstable result of aggregrate function hist #38608 (#38893)
cherry pick from #38608
2024-08-06 08:52:03 +08:00
e9bf0776d7 [fix](parquet) disable parquet page index by default #38691 (#38901)
bp #38691
2024-08-06 08:51:39 +08:00
70a518e099 [Fix](multi-catalog) Fix not throw error when call close() in hive/iceberg writer. (#38902)
## Proposed changes
[Fix] (multi-catalog) Fix not throw error when call close() in
hive/iceberg writer.

When the file writer closes(), it will sync buffer to commit. Therefore,
sometimes data is written only when close() is called, which can expose
some errors. For example, hdfs_file_writer. Therefore, this error needs
to be captured in the entire close process.
2024-08-06 08:51:12 +08:00
3b9394a8c7 [improvement](tablet scheduler) Adjust tablet sched priority to help load data succ #38528 (#38884)
cherry pick from #38528
2024-08-06 02:13:47 +08:00
0711423ee3 [Chore](pipeline) set PipelineFragmentContext::_timeout (#38890)
## Proposed changes

Now we use `query_timeout` to set a timeout value for queries. But for
pipelineX engine, Doris do not use it so each query will not end before
EOS. This PR fix it.

pick #35328

<!--Describe your changes.-->
2024-08-05 21:47:08 +08:00
9c020f9db1 [fix](fe) Fix the default value of ReplacePartitionClause.isStrictRange (#38688) (#38879) 2024-08-05 20:59:50 +08:00
ce75e6adfe [fix](group commit) Fix group commit debug log and improve performance (#38754) (#38841)
Pick https://github.com/apache/doris/pull/38754
2024-08-05 18:34:49 +08:00
0f0b0e9b37 [Feat](nereids) Support date_trunc function in partition prune (#38025) (#38849)
cherry-pick #38025 to branch-2.1
2024-08-05 18:29:10 +08:00
9d5af7febd [opt](inverted index) Optimization of the initialization process in topn (#38870)
pick https://github.com/apache/doris/pull/37722
2024-08-05 18:26:00 +08:00
40567b5d69 [fix](nereids)support group_concat with distinct and order by (#38871)
## Proposed changes

pick from master https://github.com/apache/doris/pull/38080

<!--Describe your changes.-->
2024-08-05 18:23:55 +08:00
bf1c7a1c15 [fix](clone) fix stale tablet report miss the new cloning replica #38695 (#38839)
cherry pick from #38695
2024-08-05 18:04:24 +08:00
0f69a2a47f [fix](compaction) fix mismatch between segment key and value column rows during compaction (#37960)(#38251)(#38356) (#38835)
pick master #37960 #38251 #38356
2024-08-05 16:48:08 +08:00
994c56f914 [fix](txn) fix abortTxn by label does not acquire table write lock (#38777) (#38842)
pick https://github.com/apache/doris/pull/38777
2024-08-05 16:33:20 +08:00
7d4ff34d1f [fix](regression) fix test_primary_key_simple_case (#38798) (#38844)
pick https://github.com/apache/doris/pull/38798
2024-08-05 16:32:41 +08:00
4c75fecea9 [fix](compile) be compile failed in mac due to std::max (#37238) (#38860)
cherry-pick #37238 to branch-2.1
2024-08-05 16:31:39 +08:00
bb962a8291 [minor](fix) Fix incorrect fmt arguments (#38840) (#38861)
pick #38840
2024-08-05 16:06:32 +08:00
65154f8abe [branch-2.1] (doris-future) Support auto partition name function (#38853)
cherry-pick https://github.com/apache/doris/pull/34258 to branch-2.1
2024-08-05 16:04:24 +08:00
Pxl
86ef0069ea [Feature](function) support group concat with distinct and order by (#38851)
pick from #38744 and #38776
2024-08-05 15:44:51 +08:00
5dfc5d2c77 [enhancement](querycancel) print detail message when query is cancelled (#38859)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-08-05 14:47:03 +08:00
808397e0d2 [fix](testcase) add order by to fix unstable output of passwordLeaked #38813 (#38855)
cherry pick from #38813
2024-08-05 13:51:54 +08:00
aaee1d9bbd [fix](regression) fix prepare_insert when execute prepare stmt in observer fe (#38545) (#38850)
pick https://github.com/apache/doris/pull/38545
2024-08-05 13:45:13 +08:00
de9b9d6a39 [Fix](nereids) change char(0) to char(1), varchar(0) to varchar(65533) when create table (#38427) (#38530)
cherry-pick #38427 to branch-2.1

---------

Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
2024-08-05 09:18:18 +08:00
9430b27e68 [branch-2.1][improvement](jdbc catalog) improvement some jdbc catalog properties check order (#38770)
pick (#38439)

1. Move the execution of testJdbcConnection() to checkWhenCreating
instead of the constructor
2. Move the logic of renaming lower_case_table_names to
lower_case_meta_names to setDefaultPropsIfMissing
2024-08-05 09:14:04 +08:00
607c0b82a9 [opt](serde)Optimize the filling of fixed values ​​into block columns without repeated deserialization. (#37377) (#38245) (#38810)
## Proposed changes
pick pr: #38575  and fix this pr bug :  #38245
2024-08-05 09:13:08 +08:00
2653087843 [pick](array-funcs)fix array with empty arg in be behavior (#38708)
## Proposed changes
backport: https://github.com/apache/doris/pull/36845
Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-05 09:08:28 +08:00
1b3d4b4d31 [cherry-pick](branch-21)fix operator do_projections should use local_state intermediate_projections (#38612) (#38765)
## Proposed changes

cherry-pick from master https://github.com/apache/doris/pull/38612

<!--Describe your changes.-->
2024-08-05 09:07:16 +08:00