01acd60e2b
[Pick][Improment]Add workload group query detail ( #40050 ) ( #41594 )
...
pick #40050
2024-10-11 09:32:33 +08:00
6dddd4c499
[function](cast)Make string casting to integers more like MySQL's beh… ( #41541 )
...
…avior (#38847 )
https://github.com/apache/doris/pull/38847
## Proposed changes
There are two issues here. First, the results of casting are
inconsistent between FE and BE .
```
FE
mysql [(none)]>select cast('3.000' as int);
+----------------------+
| cast('3.000' as INT) |
+----------------------+
| 3 |
+----------------------+
mysql [(none)]>set debug_skip_fold_constant = true;
BE
mysql [(none)]>select cast('3.000' as int);
+----------------------+
| cast('3.000' as INT) |
+----------------------+
| NULL |
+----------------------+
```
The second issue is that casting on BE converts '3.0' to null. Here, the
casting logic for FE and BE has been unified
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
---------
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com >
2024-10-11 09:32:00 +08:00
4c9ebbb3b9
[fix](cloud) cloud group commit should skip repaly wal if label is already used and the txn state is committed or visible ( #41262 ) ( #41461 )
...
pick https://github.com/apache/doris/pull/41262
2024-10-10 22:27:04 +08:00
f2ba1f2fb3
[bugfix](segmentload) should remove segment from segment cache if load segment failed ( #41608 ) ( #41660 )
2024-10-10 19:40:22 +08:00
0fb42d3a48
[Enhancement](tvf)catalog tvf implements user permission checks and hides sensitive information ( #41497 ) ( #41604 )
...
bp #41497
before #21790
## Proposed changes
This PR unifies the duplicate parts of `catalog tvf` and `show
catalogs`, adds permission check when querying `catalog tvf`, and hides
sensitive information.
2024-10-10 17:55:40 +08:00
1db0aef9b7
[feature](array_agg) support array_agg with param is array/map/struct… ( #41651 )
...
… (#40697 )
this pr we support array_agg function support param with array map
struct type
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-10 17:54:54 +08:00
3120bfb6e3
[fix](pipelinex) fix fragment instance progress reports (part 2) ( #40694 ) ( #41641 )
...
backport #40694
2024-10-10 17:49:41 +08:00
30492a2438
[opt](load) print more detailed log when stream load finished #41398 ( #41639 )
...
cherry pick from #41398
2024-10-10 17:47:48 +08:00
d32688e091
[Enhancement](multi-catalog) Set hdfs native client logger to glog and redirect jvm stdout/stderr logger to jni.log. ( #41633 )
...
Backport #39540 .
Co-authored-by: Mingyu Chen <morningman@163.com >
2024-10-10 17:47:21 +08:00
070bd6cf99
[fix](jdbc) fix Unknown command(27) ( #41621 ) ( #41637 )
...
pick https://github.com/apache/doris/pull/41621
2024-10-10 17:46:45 +08:00
a26079c09d
[Opt](load) Optimize the error messages of -235 and -238 for loading #41048 ( #41638 )
...
cherry pick from #41048
2024-10-10 14:20:52 +08:00
cd26713c26
[fix](Nereids) could not parse date/datetime with blank + zone ( #41394 ) ( #41620 )
...
pick from master #41394
for example:
2008-08-08 20:08:08 +08:00 parse failed because the blank before +08:00
2024-10-10 14:03:41 +08:00
33fad04341
[opt](Nereids) use 1 instead narrowest column when do column pruning ( #41548 ) ( #41627 )
...
pick from master #41548
2024-10-10 14:02:23 +08:00
eb348ec10e
[improve](binlog) Filter the truncated partitions #41611 ( #41630 )
...
cherry pick from #41611
2024-10-10 11:59:27 +08:00
aa541fddf9
[fix](load) disable num segments check in compatibility mode ( #41053 ) ( #41552 )
...
backport #41053
2024-10-10 11:20:16 +08:00
6a3dc98361
[fix](Nereids) merge request properties map of group should compare cost ( #40819 ) ( #41615 )
...
pick from master #40819
when do merge, we should update target requestPropertiesMap ONLY IF the
cost of source's request property lower than target one. Otherwise, the
requestPropertiesMap will not sync with lowestCostTable. Then, we will
get wrong output property when get the final plan.
2024-10-10 11:16:00 +08:00
032c00e3c9
[branch-2.1](create table) show failed detail msg #41463 ( #41544 )
...
cherry-pick: #41463
2024-10-10 11:15:02 +08:00
15bdfa446c
[case](mtmv)fix alter job case ( #41326 ) ( #41500 )
...
pick: https://github.com/apache/doris/pull/41326
2024-10-10 00:11:50 +08:00
a45dc8796a
[fix](Nereids) simplify decimal comparison wrong when cast to smaller scale ( #41151 ) ( #41618 )
...
pick from master #41151
2024-10-09 23:03:01 +08:00
649cefd70f
[opt](Nereids) forbid distribute under project and filter ( #39812 ) ( #41622 )
...
pick from master #39812
2024-10-09 23:02:06 +08:00
1d2e2adb74
[fix](parser) should not use selectHint in any place ( #41260 ) ( #41619 )
...
pick from master #41260
because all comment has been redirect to channel 2, we should not use
hint in any place. selectHint only use to parse hint.
2024-10-09 23:00:04 +08:00
ade86c0600
Revert "[branch-2.1][improvement](jdbc catalog) Optimize JdbcCatalog case mapping stability" ( #41588 )
...
Reverts apache/doris#41330
2024-10-09 20:51:58 +08:00
e218fd2314
[Fix](inverted index) add DATEV2 and DATETIMEV2 for inverted index reader #41565 ( #41579 )
...
cherry pick from #41565
2024-10-09 15:32:41 +08:00
31b506c8cc
[Enhancement](inverted index) return OK instead of not supported in expr evaluate_inverted_index #41567 ( #41578 )
...
cherry pick from #41567
2024-10-09 15:14:38 +08:00
0185f8069f
[fix](crash) fix be crash because of int overflow ( #41554 ) ( #41568 )
2024-10-09 14:20:55 +08:00
9fe77b335c
[Enhancement](inverted index) apply inverted index when has any #41547 ( #41584 )
...
cherry pick from #41547
2024-10-09 14:13:38 +08:00
5a6517ba45
[test](inverted index) fix test case for no need read data #41564 ( #41583 )
...
cherry pick from #41564
2024-10-09 14:03:02 +08:00
b6c0603e0d
[fix](mtmv)Fix show mtmv time should wrapped in double quotation mark… ( #41501 )
...
pick: https://github.com/apache/doris/pull/41419
2024-10-09 12:19:28 +08:00
308700f0ca
[fix](test) fix unstable test_export_external_table cases ( #41523 ) ( #41570 )
...
bp #41523
2024-10-09 11:53:22 +08:00
25684f487b
[2.1][improvement](jdbc catalog) Improve JdbcClientException to accommodate various identifier formats ( #41530 )
...
pick (#40931 )
In some cases, JDBC returns exceptions with various identifiers that
cannot be formatted correctly, such as `%`. This PR optimizes this.
2024-10-09 10:32:41 +08:00
a0aed77218
[cherry-pick](branch2.1) fix hudi jni scanner ( #41566 )
...
pick from https://github.com/apache/doris/pull/41316
2024-10-09 10:31:50 +08:00
afb477c66d
[Fix](inverted index) Fix wrong need read data opt when enable_common_expr_pushdown is disabled #40689 ( #41562 )
...
cherry pick from #40689
2024-10-08 22:12:10 +08:00
c24ff2ff81
[fix](upgrade) fix version check failure of window_funnel when upgrading ( #41542 )
...
## Proposed changes
Issue Number: close #xxx
Fix fix version check failure of window_funnel when upgrading from 2.1.6
and higher version to latest branch 2.1.
```
02:49:13 F20240930 02:47:48.546983 7581 block.cpp:89] Check failed: BeExecVersionManager::check_be_exec_version(be_exec_version)
02:49:13 *** Check failure stack trace: ***
02:49:13 @ 0x564640041856 google::LogMessage::SendToLog()
02:49:13 @ 0x56464003e2a0 google::LogMessage::Flush()
02:49:13 @ 0x564640042099 google::LogMessageFatal::~LogMessageFatal()
02:49:13 @ 0x56463922d106 doris::vectorized::Block::deserialize()
02:49:13 @ 0x5646390a82bf doris::vectorized::WindowFunnelState<>::read()
02:49:13 @ 0x5646390a6889 doris::vectorized::IAggregateFunctionDataHelper<>::deserialize_and_merge()
02:49:13 @ 0x5646390acdc3 doris::vectorized::IAggregateFunctionHelper<>::deserialize_and_merge_from_column_range()
02:49:13 @ 0x56463fa77152 doris::pipeline::AggSinkLocalState::_merge_without_key()
02:49:13 @ 0x56463fa9d114 doris::pipeline::AggSinkLocalState::Executor<>::execute()
02:49:13 @ 0x56463fa78569 doris::pipeline::AggSinkOperatorX::sink()
02:49:13 @ 0x564640013296 doris::pipeline::PipelineXTask::execute()
02:49:13 @ 0x56464001d41c doris::pipeline::TaskScheduler::_do_work()
02:49:13 @ 0x56463663e078 doris::ThreadPool::dispatch_thread()
02:49:13 @ 0x564636634901 doris::Thread::supervise_thread()
02:49:13 @ 0x7fb64cf58ac3 (unknown)
02:49:13 @ 0x7fb64cfea850 (unknown)
02:49:13 @ (nil) (unknown)
02:49:13 *** Query id: b0cd194940184766-961c310e833e92b1 ***
02:49:13 *** is nereids: 1 ***
02:49:13 *** tablet id: 0 ***
02:49:13 *** Aborted at 1727635668 (unix time) try "date -d @1727635668" if you are using GNU date ***
02:49:13 *** Current BE git commitID: 653e315ba5 ***
02:49:13 *** SIGABRT unknown detail explain (@0x1648) received by PID 5704 (TID 7581 OR 0x7fb354a9a640) from PID 5704; stack trace: ***
02:49:13 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/common/signal_handler.h:421
02:49:13 1# 0x00007FB64CF06520 in /lib/x86_64-linux-gnu/libc.so.6
02:49:13 2# pthread_kill at ./nptl/pthread_kill.c:89
02:49:13 3# raise at ../sysdeps/posix/raise.c:27
02:49:13 4# abort at ./stdlib/abort.c:81
02:49:13 5# 0x000056464004C06D in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 6# 0x000056464003E76A in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 7# google::LogMessage::SendToLog() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 8# google::LogMessage::Flush() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 10# doris::vectorized::Block::deserialize(doris::PBlock const&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/core/block.cpp:113
02:49:13 11# doris::vectorized::WindowFunnelState<(doris::vectorized::TypeIndex)14, long>::read(doris::vectorized::BufferReadable&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function_window_funnel.h:363
02:49:13 12# doris::vectorized::IAggregateFunctionDataHelper<doris::vectorized::WindowFunnelState<(doris::vectorized::TypeIndex)14, long>, doris::vectorized::AggregateFunctionWindowFunnel<(doris::vectorized::TypeIndex)14, long> >::deserialize_and_merge(char*, char*, doris::vectorized::BufferReadable&, doris::vectorized::Arena*) const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function.h:517
02:49:13 13# doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionNullVariadicInline<doris::vectorized::AggregateFunctionWindowFunnel<(doris::vectorized::TypeIndex)14, long>, false> >::deserialize_and_merge_from_column_range(char*, doris::vectorized::IColumn const&, unsigned long, unsigned long, doris::vectorized::Arena*) const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function.h:465
02:49:13 14# doris::pipeline::AggSinkLocalState::_merge_without_key(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:389
02:49:13 15# doris::pipeline::AggSinkLocalState::Executor<true, true>::execute(doris::pipeline::AggSinkLocalState*, doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/exec/aggregation_sink_operator.h:73
02:49:13 16# doris::pipeline::AggSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:744
02:49:13 17# doris::pipeline::PipelineXTask::execute(bool*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:332
02:49:13 18# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:347
02:49:13 19# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:499
02:49:13 21# start_thread at ./nptl/pthread_create.c:442
02:49:13 22# 0x00007FB64CFEA850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
02:49:13 [2024-09-30 02:49:13,147 __main__:796] [INFO]: 172.20.50.73 last coredump sql: 2024-09-30 02:48:18,328 [query] Query b0cd194940184766-961c310e833e92b1 1 times with new query id: 2e0e00de0e7548dd-95f9abc9d8d11c3a
```
2024-10-08 17:13:33 +08:00
ca59c8537c
[chore](query err) fix cutting err msg incorrectly #41456 ( #41538 )
...
cherry pick from #41456
2024-10-08 16:09:15 +08:00
3355ebe5e7
[enhancement](delete) Using insert timeout session var to control delete job timeout ( #41063 ) ( #41475 )
...
## Proposed changes
pick: #41063
2024-10-08 12:03:30 +08:00
cb24ccc112
[bugfix](brpc) Should use status to generate protobuf message, because it will encoding Backend Info ( #41515 ) ( #41522 )
...
Should use status to generate protobuf message, because it will encoding
Backend Info
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-04 23:03:55 +08:00
d33a3eb1c5
[cherry-pick](branch-2.1) Pick "[Fix](LZ4 compression) Fix wrong LZ4 compression max input size limit ( #41239 )" ( #41505 )
...
## Proposed changes
LZ4 compression max supported value is LZ4_MAX_INPUT_SIZE, which is
0x7E000000(2,113,929,216 bytes). Doris use wrong max size INT_MAX, which
is 2,147,483,647, to check. If input data size is between this two size,
then it can pass the check but LZ4 compression will fail.
This PR fix it.
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-01 22:43:11 +08:00
806fe98a93
[2.1][fix](oracle scan) Fix performance issues caused by version judgment ( #41509 )
...
pick (#41407 )
2024-10-01 21:04:22 +08:00
4f81fc474c
[bugfix](paimon)Get the file format by file name ( #41020 ) ( #41487 )
...
bp #41020
2024-09-30 15:46:13 +08:00
78b95e7595
[chore](configuration) Change be jvm default max heap size. ( #41483 )
...
## Proposed changes
Backport #41281
2024-09-30 15:41:25 +08:00
98a1311aa2
[Opt](scanner-scheduler) Opt scanner scheduler starvation issue. ( #41484 )
...
## Proposed changes
Backport #40641
2024-09-30 15:40:20 +08:00
03ef90b96e
[regression-test](feature) support skip non-concurrent ( #41493 )
...
…1021)
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-30 12:47:18 +08:00
b5e94b65c0
[fix](mtmv) Generate mtmv cache should use ADMIN user, and rewritten plan should not check privilege ( #40374 ) ( #41450 )
...
## Proposed changes
pr: https://github.com/apache/doris/pull/40374
commitId: f3d92e48
2024-09-30 09:59:54 +08:00
d5dccac270
[enhancement](test) Throw exception explicitly in forward query test ( #40742 ) ( #41470 )
...
## Proposed changes
pick: #40742
2024-09-30 09:54:24 +08:00
70630f244e
[fix](regression-test) Add varchar column as distribution key ( #40791 ) ( #41471 )
...
pick: #40791
2024-09-30 09:53:18 +08:00
2b9c963edb
[fix](scanner) Check query status when iterating through rowsets and segments #41363 ( #41452 )
...
cherry pick from #41363
2024-09-30 09:49:46 +08:00
b7db357847
[test](inverted index) refine test_ignore_above case, add compound query sql #40355 ( #41445 )
...
cherry pick from #40355
2024-09-30 07:04:08 +08:00
a7b88af861
[bugfix](external)Memory leak problem for external table with insert operation for 2.1 ( #40440 ) ( #41479 )
...
bp: #40440
2024-09-29 23:09:25 +08:00
9864324286
[enhancement](err-msg) Clarify error msg for forward queries ( #40802 ) ( #41474 )
...
## Proposed changes
pick: #40802
2024-09-29 22:23:13 +08:00
676c4e3b43
[Fix](inverted index) fix comparison for string after refactor compound #40338 ( #41441 )
...
cherry pick from #40338
2024-09-29 22:10:01 +08:00
