64d0c55e1d
[fix](decimal256) support decimal256 for many functions ( #42136 ) ( #42353 )
...
BP #42136
2024-10-24 10:52:28 +08:00
1db0aef9b7
[feature](array_agg) support array_agg with param is array/map/struct… ( #41651 )
...
… (#40697 )
this pr we support array_agg function support param with array map
struct type
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-10 17:54:54 +08:00
c24ff2ff81
[fix](upgrade) fix version check failure of window_funnel when upgrading ( #41542 )
...
## Proposed changes
Issue Number: close #xxx
Fix fix version check failure of window_funnel when upgrading from 2.1.6
and higher version to latest branch 2.1.
```
02:49:13 F20240930 02:47:48.546983 7581 block.cpp:89] Check failed: BeExecVersionManager::check_be_exec_version(be_exec_version)
02:49:13 *** Check failure stack trace: ***
02:49:13 @ 0x564640041856 google::LogMessage::SendToLog()
02:49:13 @ 0x56464003e2a0 google::LogMessage::Flush()
02:49:13 @ 0x564640042099 google::LogMessageFatal::~LogMessageFatal()
02:49:13 @ 0x56463922d106 doris::vectorized::Block::deserialize()
02:49:13 @ 0x5646390a82bf doris::vectorized::WindowFunnelState<>::read()
02:49:13 @ 0x5646390a6889 doris::vectorized::IAggregateFunctionDataHelper<>::deserialize_and_merge()
02:49:13 @ 0x5646390acdc3 doris::vectorized::IAggregateFunctionHelper<>::deserialize_and_merge_from_column_range()
02:49:13 @ 0x56463fa77152 doris::pipeline::AggSinkLocalState::_merge_without_key()
02:49:13 @ 0x56463fa9d114 doris::pipeline::AggSinkLocalState::Executor<>::execute()
02:49:13 @ 0x56463fa78569 doris::pipeline::AggSinkOperatorX::sink()
02:49:13 @ 0x564640013296 doris::pipeline::PipelineXTask::execute()
02:49:13 @ 0x56464001d41c doris::pipeline::TaskScheduler::_do_work()
02:49:13 @ 0x56463663e078 doris::ThreadPool::dispatch_thread()
02:49:13 @ 0x564636634901 doris::Thread::supervise_thread()
02:49:13 @ 0x7fb64cf58ac3 (unknown)
02:49:13 @ 0x7fb64cfea850 (unknown)
02:49:13 @ (nil) (unknown)
02:49:13 *** Query id: b0cd194940184766-961c310e833e92b1 ***
02:49:13 *** is nereids: 1 ***
02:49:13 *** tablet id: 0 ***
02:49:13 *** Aborted at 1727635668 (unix time) try "date -d @1727635668" if you are using GNU date ***
02:49:13 *** Current BE git commitID: 653e315ba5 ***
02:49:13 *** SIGABRT unknown detail explain (@0x1648) received by PID 5704 (TID 7581 OR 0x7fb354a9a640) from PID 5704; stack trace: ***
02:49:13 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/common/signal_handler.h:421
02:49:13 1# 0x00007FB64CF06520 in /lib/x86_64-linux-gnu/libc.so.6
02:49:13 2# pthread_kill at ./nptl/pthread_kill.c:89
02:49:13 3# raise at ../sysdeps/posix/raise.c:27
02:49:13 4# abort at ./stdlib/abort.c:81
02:49:13 5# 0x000056464004C06D in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 6# 0x000056464003E76A in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 7# google::LogMessage::SendToLog() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 8# google::LogMessage::Flush() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 10# doris::vectorized::Block::deserialize(doris::PBlock const&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/core/block.cpp:113
02:49:13 11# doris::vectorized::WindowFunnelState<(doris::vectorized::TypeIndex)14, long>::read(doris::vectorized::BufferReadable&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function_window_funnel.h:363
02:49:13 12# doris::vectorized::IAggregateFunctionDataHelper<doris::vectorized::WindowFunnelState<(doris::vectorized::TypeIndex)14, long>, doris::vectorized::AggregateFunctionWindowFunnel<(doris::vectorized::TypeIndex)14, long> >::deserialize_and_merge(char*, char*, doris::vectorized::BufferReadable&, doris::vectorized::Arena*) const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function.h:517
02:49:13 13# doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionNullVariadicInline<doris::vectorized::AggregateFunctionWindowFunnel<(doris::vectorized::TypeIndex)14, long>, false> >::deserialize_and_merge_from_column_range(char*, doris::vectorized::IColumn const&, unsigned long, unsigned long, doris::vectorized::Arena*) const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function.h:465
02:49:13 14# doris::pipeline::AggSinkLocalState::_merge_without_key(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:389
02:49:13 15# doris::pipeline::AggSinkLocalState::Executor<true, true>::execute(doris::pipeline::AggSinkLocalState*, doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/exec/aggregation_sink_operator.h:73
02:49:13 16# doris::pipeline::AggSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:744
02:49:13 17# doris::pipeline::PipelineXTask::execute(bool*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:332
02:49:13 18# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:347
02:49:13 19# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:499
02:49:13 21# start_thread at ./nptl/pthread_create.c:442
02:49:13 22# 0x00007FB64CFEA850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
02:49:13 [2024-09-30 02:49:13,147 __main__:796] [INFO]: 172.20.50.73 last coredump sql: 2024-09-30 02:48:18,328 [query] Query b0cd194940184766-961c310e833e92b1 1 times with new query id: 2e0e00de0e7548dd-95f9abc9d8d11c3a
```
2024-10-08 17:13:33 +08:00
bf3d4240be
[fix](window_func) fix bug of agg function used in window function and add many test cases ( #40678 ) ( #41328 )
...
## Proposed changes
Issue Number: close #xxx
BP #40678
2024-09-26 22:50:34 +08:00
a11fd62043
[fix](window function) Fix illegal frame range ( #41147 ) ( #41305 )
...
pick #41147
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# 0x00007F591D573520 in /lib/x86_64-linux-gnu/libc.so.6
2# pthread_kill at ./nptl/pthread_kill.c:89
3# raise at ../sysdeps/posix/raise.c:27
4# abort at ./stdlib/abort.c:81
5# _nl_load_domain at ./intl/loadmsgcat.c:1177
6# 0x00007F591D56AE96 in /lib/x86_64-linux-gnu/libc.so.6
7# doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false,
false, false, DefaultMemoryAllocator>, 16ul, 15ul>::operator[](long)
const at
/home/zcp/repo_center/doris_master/doris/be/src/vec/common/pod_array.h:365
8# doris::vectorized::ColumnNullable::is_null_at(unsigned long) const at
/home/zcp/repo_center/doris_master/doris/be/src/vec/columns/column_nullable.h:158
9#
doris::vectorized::ReaderFirstAndLastData<doris::vectorized::ColumnVector<double>,
true, true, false>::insert_result_into(doris::vectorized::IColumn&)
const at
/home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function_reader_first_last.h:125
10# doris::pipeline::AnalyticLocalState::_insert_result_info(long) in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
11# std::_Function_handler<void (long), std::_Bind_result<void, void
(doris::pipeline::AnalyticLocalState::*(doris::pipeline::AnalyticLocalState*,
std::_Placeholder<1>))(long)> >::_M_invoke(std::_Any_data const&,
long&&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
12# std::function<void (long)>::operator()(long) const at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560
13# doris::pipeline::AnalyticLocalState::_get_next_for_rows(unsigned
long) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 14#
std::enable_if<is_invocable_r_v<doris::Status, doris::Status
(doris::pipeline::AnalyticLocalState::*&)(unsigned long),
doris::pipeline::AnalyticLocalState*&, unsigned long>,
doris::Status>::type std::__invoke_r<doris::Status, doris::Status
(doris::pipeline::AnalyticLocalState::*&)(unsigned long),
doris::pipeline::AnalyticLocalState*&, unsigned long>(doris::Status
(doris::pipeline::AnalyticLocalState::*&)(unsigned long),
doris::pipeline::AnalyticLocalState*&, unsigned long&&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114
15# doris::Status std::_Bind_result<doris::Status, doris::Status
(doris::pipeline::AnalyticLocalState::*(doris::pipeline::AnalyticLocalState*,
std::_Placeholder<1>))(unsigned long)>::__call<doris::Status, unsigned
long&&, 0ul, 1ul>(std::tuple<unsigned long&&>&&, std::_Index_tuple<0ul,
1ul>) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:570
16# std::_Function_handler<doris::Status (unsigned long),
std::_Bind_result<doris::Status, doris::Status
(doris::pipeline::AnalyticLocalState::*(doris::pipeline::AnalyticLocalState*,
std::_Placeholder<1>))(unsigned long)> >::_M_invoke(std::_Any_data
const&, unsigned long&&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
17# std::function<doris::Status (unsigned long)>::operator()(unsigned
long) const at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560
18#
doris::pipeline::AnalyticSourceOperatorX::get_block(doris::RuntimeState*,
doris::vectorized::Block*, bool*) in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
19#
doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*,
doris::vectorized::Block*, bool*) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/operator.cpp:322
20# doris::pipeline::PipelineTask::execute(bool*) in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
21# doris::pipeline::TaskScheduler::_do_work(unsigned long) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/task_scheduler.cpp:138
22# doris::ThreadPool::dispatch_thread() in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
23# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 24#
start_thread at ./nptl/pthread_create.c:442
25# 0x00007F591D657850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
2024-09-26 09:55:33 +08:00
ed76b0c5f5
[fix](compile) failed on MacOS because size_t is not uint64_t on MacOS ( #39296 ) ( #40720 )
...
pick #39296 to branch-2.1
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com >
2024-09-12 18:47:59 +08:00
c1abaa4679
[Bug](map) fix wrong result on map_agg with streaming agg ( #40471 )
...
pick from #40454
2024-09-06 19:29:38 +08:00
7e27bb1ae6
[fix](window_funnel) fix wrong result of fixed mode ( #40460 )
...
BP #40459
2024-09-06 09:48:24 +08:00
db0724dfe0
[Fix-2.1](function) fix function covar core for not null input ( #39943 )
...
## Proposed changes
Issue Number: close #xxx
add testcases like:
```groovy
qt_notnull1 "select covar_samp(non_nullable(x), non_nullable(y)) from test_covar_samp"
qt_notnull2 "select covar_samp(x, non_nullable(y)) from test_covar_samp"
qt_notnull3 "select covar_samp(non_nullable(x), y) from test_covar_samp"
```
before they will all coredump in 2.1
2024-08-27 08:39:47 +08:00
460605ae3c
[branch-2.1] pick some prs ( #39860 )
...
## Proposed changes
Issue Number: close #xxx
https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
2024-08-24 17:26:42 +08:00
8cf6c6a2b5
[fix](agg function) incorrect result of map agg( #39743 ) ( #39854 )
...
## Proposed changes
pick #39743
2024-08-24 09:19:52 +08:00
ba7baa7e6b
[fix](window_funnel) fix upgrading problem caused by behaviour change of window_funnel ( #39766 )
...
## Proposed changes
Issue Number: close #xxx
For the latest published 2.1 version `2.1.5`:
```
max_be_exec_version=4;
AGG_FUNCTION_NEW=2;
```
and `branch-2.1`:
```
max_be_exec_version=5;
AGG_FUNCTION_NEW=2;
```
It will cause problem when upgrading.
This PR fix the problem, set `AGG_FUNCTION_NEW` to `5`.
2024-08-22 17:26:51 +08:00
1e30d4ebaf
Revert "[Bug](compatibility) fix window funnel function coredump when upgrade" ( #39681 )
...
Reverts apache/doris#39646
2024-08-21 14:47:27 +08:00
75eded04d7
[Bug](compatibility) fix window funnel function coredump when upgrade ( #39646 )
...
## Proposed changes
this PR https://github.com/apache/doris/pull/39270 have change the agg
of window funnel
and max_be_exec_version is update to 5, in order to compatibility of the
agg function when upgrade.
<!--Describe your changes.-->
2024-08-21 08:46:50 +08:00
021678c7c3
[fix](window_funnel) fix wrong result of window_funnel #38954 ( #39270 )
...
## Proposed changes
BP #38954
2024-08-16 09:59:31 +08:00
b26af32934
[fix](function) fix error return type in corr(float32,float32) ( #39251 ) ( #39350 )
...
https://github.com/apache/doris/pull/39251
```
mysql [test11]>select corr(cast(x as float),cast(y as float)) from test_corr;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]column_type not match data_types in agg node, column_type=Nullable(Float64), data_types=Nullable(Float32),column name=
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-14 18:47:14 +08:00
efdd75f286
[fix](function) stddev with DecimalV2 type will result in an error (#… ( #39072 )
...
…38731)
https://github.com/apache/doris/pull/38731
The stddev function has a separate implementation for the DecimalV2
type, but there are issues with the implementation. Given that there is
almost no existing data for DecimalV2, it will be removed here. For be,
upgrading to this situation will result in an error directly.
```
SELECT STDDEV(data) FROM DECIMALV2_10_0_DATA;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Agg Function stddev(decimal(10,0)) is not implemented
```
After removing DecimalV2, parameters of type DecimalV2 will be converted
to double for calculations.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-08 17:53:17 +08:00
c7b59b38ef
[fix](hist) Fix unstable result of aggregrate function hist #38608 ( #38893 )
...
cherry pick from #38608
2024-08-06 08:52:03 +08:00
86ef0069ea
[Feature](function) support group concat with distinct and order by ( #38851 )
...
pick from #38744 and #38776
2024-08-05 15:44:51 +08:00
d7e84b7ee3
[Enchancement](bitmap) optimize bitmap deserialize and remove some unused code ( #37623 )
...
## Proposed changes
pick from #35789
2024-07-16 11:21:54 +08:00
e64f2997e9
[fix](function) fix core when input not null array in foreach functio… ( #37798 )
...
## Proposed changes
https://github.com/apache/doris/pull/37349
error code
```C++
return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true,
nested_function);
```
"transform_arguments is an internal type of array. All internal types of
the array are null, so an array that is not null was mistakenly treated
as a null array."
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-16 10:57:11 +08:00
90a4dd09f3
[Fix](func) CoreDump and Result Error in percentile function ( #36647 )
...
cherry pick #36643
2024-06-21 23:42:45 +08:00
c794ea18c8
[fix](multi-catalog)put java udf to custom lib ( #35984 )
...
bp #34990
2024-06-06 22:54:24 +08:00
596fb6f327
[improve](ub) fix some runtime error of ubsan when downcast ( #35343 )
...
those code could work well, but it will be report some runtime error under UBSAN,
so refactor it to let's ubsan could running happy.
2024-05-27 15:27:43 +08:00
8fdfbcb3c4
Revert "[Opt](func) opt the percentile func performance ( #34373 ) ( #34416 )"
...
This reverts commit 509ae425e416b4779ae94eab9c2b21f9850e03c3.
2024-05-07 07:23:48 +08:00
509ae425e4
[Opt](func) opt the percentile func performance ( #34373 ) ( #34416 )
2024-05-06 20:10:35 +08:00
970d0c80df
[Improvement](agg) Improve count distinct distribute keys ( #33167 )
2024-04-27 02:29:33 +08:00
10e098845d
[fix](compile) fix two compile errors on MacOS ( #33834 ) ( #34149 )
2024-04-26 17:02:44 +08:00
47ded2c6a0
Revert "[fix](compile) fix two compile errors on MacOS ( #33834 ) ( #34005 )"
...
This reverts commit 743fb62a2c42cc5cc662583c235f7336d5e6ddef.
2024-04-26 00:55:21 +08:00
743fb62a2c
[fix](compile) fix two compile errors on MacOS ( #33834 ) ( #34005 )
2024-04-25 19:39:35 +08:00
25358564ca
[Fix](compile) Fix gcc compile on master ( #33864 )
...
This is imported by #33511 . wrongly used
ColumnStr<T> ();
which violate C++20 standard(see https://wg21.cmeerw.net/cwg/issue2237 ) but still supported by clang up until now(see llvm/llvm-project#58112 )
2024-04-19 23:41:37 +08:00
1300317723
[Exec](join) Support column string64 to avoid join failed in string size overflow the uint32 ( #33511 ) ( #33850 )
2024-04-18 19:43:08 +08:00
3096150d1b
[feature](agg) support aggregate function group_array_intersect ( #33265 )
2024-04-17 23:42:13 +08:00
8fd6d4c41b
[Chore](build) add -Wconversion and remove some unused code ( #33127 )
...
add -Wconversion and remove some unused code
2024-04-10 15:26:08 +08:00
bb8bc75af4
[feature](agg) add aggregate function sum0 ( #32541 )
2024-04-10 11:34:29 +08:00
e3bd2311b1
[Performance](exec) replace SipHash in function by XXHash ( #32919 )
2024-04-10 11:34:29 +08:00
0a447273f0
[Bug](pipelineX) fix memleak in stream agg ( #32862 )
...
* [Bug](pipelineX) fix memleak in stream agg
* change by review
2024-03-27 09:21:21 +08:00
a55b5ea9ca
[bug](udaf) fix memory leak in the java udaf ( #32630 )
...
fix memory leak in the java udaf
2024-03-24 08:06:13 +08:00
2cb652a7fa
[FIX](compile)fix for gcc compile ( #32508 )
...
* fix for gcc compile
2024-03-22 08:52:16 +08:00
b92a764665
[feature](function) Support for aggregate function foreach combiner for some error function ( #31913 )
...
Support for aggregate function foreach combiner for some error function
2024-03-21 14:07:49 +08:00
7a0b591b8f
[FIX](array_agg) fix array agg with other agg function ( #32387 )
...
fix array agg with other agg function
2024-03-21 14:07:23 +08:00
5ceccb5ba5
[fix](compatibility) should enable windown funnel mode from 2.0 ( #32284 )
2024-03-16 20:56:16 +08:00
bbdce3eb5e
[fix](jdbc catalog) fix jdbc-connector coredump as get env return nullptr ( #32217 )
2024-03-14 16:05:08 +08:00
847ec368be
[Fix](smooth-upgrade) Fix incompatibility when upgrade from 2.0 to 2.1 ( #32220 )
2024-03-14 11:23:05 +08:00
93d298d34a
[fix](agg) wrong result of two or more map_agg functions in query ( #31928 )
2024-03-09 19:45:03 +08:00
4f174c4fb9
[feature](function) Support for aggregate function foreach combiner ( #31526 )
2024-03-06 13:08:30 +08:00
d70776af55
[feature](agg-func) support covar and covar_samp function ( #30983 )
2024-02-18 11:50:17 +08:00
e68019c10a
[Function](Exec) Support windows function cume_dist ( #30997 )
2024-02-16 10:16:40 +08:00
40e1326bc9
[feature](window-func) support percent_rank window function ( #30926 )
2024-02-16 10:12:25 +08:00
2bb477bae7
[feature](agg-func) support corr function #30822
2024-02-16 10:12:24 +08:00
