6dddd4c499
[function](cast)Make string casting to integers more like MySQL's beh… ( #41541 )
...
…avior (#38847 )
https://github.com/apache/doris/pull/38847
## Proposed changes
There are two issues here. First, the results of casting are
inconsistent between FE and BE .
```
FE
mysql [(none)]>select cast('3.000' as int);
+----------------------+
| cast('3.000' as INT) |
+----------------------+
| 3 |
+----------------------+
mysql [(none)]>set debug_skip_fold_constant = true;
BE
mysql [(none)]>select cast('3.000' as int);
+----------------------+
| cast('3.000' as INT) |
+----------------------+
| NULL |
+----------------------+
```
The second issue is that casting on BE converts '3.0' to null. Here, the
casting logic for FE and BE has been unified
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
---------
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com >
2024-10-11 09:32:00 +08:00
0fb42d3a48
[Enhancement](tvf)catalog tvf implements user permission checks and hides sensitive information ( #41497 ) ( #41604 )
...
bp #41497
before #21790
## Proposed changes
This PR unifies the duplicate parts of `catalog tvf` and `show
catalogs`, adds permission check when querying `catalog tvf`, and hides
sensitive information.
2024-10-10 17:55:40 +08:00
1db0aef9b7
[feature](array_agg) support array_agg with param is array/map/struct… ( #41651 )
...
… (#40697 )
this pr we support array_agg function support param with array map
struct type
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-10 17:54:54 +08:00
33fad04341
[opt](Nereids) use 1 instead narrowest column when do column pruning ( #41548 ) ( #41627 )
...
pick from master #41548
2024-10-10 14:02:23 +08:00
a45dc8796a
[fix](Nereids) simplify decimal comparison wrong when cast to smaller scale ( #41151 ) ( #41618 )
...
pick from master #41151
2024-10-09 23:03:01 +08:00
649cefd70f
[opt](Nereids) forbid distribute under project and filter ( #39812 ) ( #41622 )
...
pick from master #39812
2024-10-09 23:02:06 +08:00
308700f0ca
[fix](test) fix unstable test_export_external_table cases ( #41523 ) ( #41570 )
...
bp #41523
2024-10-09 11:53:22 +08:00
afb477c66d
[Fix](inverted index) Fix wrong need read data opt when enable_common_expr_pushdown is disabled #40689 ( #41562 )
...
cherry pick from #40689
2024-10-08 22:12:10 +08:00
4f81fc474c
[bugfix](paimon)Get the file format by file name ( #41020 ) ( #41487 )
...
bp #41020
2024-09-30 15:46:13 +08:00
b7db357847
[test](inverted index) refine test_ignore_above case, add compound query sql #40355 ( #41445 )
...
cherry pick from #40355
2024-09-30 07:04:08 +08:00
d659750fd9
[pick](Serde-2.1) fix variant serde may lost num_rows when subcolumns empty ( #41438 )
...
serialization object with empty subcolumns may lost num_rows, so need to
record num_rows and set back num_rows in serdes
backport #38413
2024-09-29 09:45:37 +08:00
727f0374be
[Refactor](inverted index) refactor inverted index compound predicates evaluate logic #38908 ( #41385 )
...
cherry pick from #38908
2024-09-29 09:19:17 +08:00
0b4552f74b
[cherry-pick](branch-2.1) pick hive text write from master ( #40537 )
...
## Proposed changes
pick prs:
https://github.com/apache/doris/pull/38549
https://github.com/apache/doris/pull/40183
https://github.com/apache/doris/pull/40315
---------
Co-authored-by: Calvin Kirs <kirs@apache.org >
2024-09-27 20:57:07 +08:00
82228358b9
[Fix](nereids) fix create view with nullable column ( #41234 ) ( #41393 )
...
cherry-pick from master #41234
2024-09-27 19:13:54 +08:00
0c51ee26ea
[fix](function) add time type in conditional-functions ( #41270 ) ( #41379 )
...
## Proposed changes
https://github.com/apache/doris/pull/41270
<!--Describe your changes.-->
2024-09-27 17:19:54 +08:00
eb13cd4154
[branch-2.1] Picks "[Fix](partial update) Fix __DORIS_SEQUENCE_COL__ is not set for newly inserted rows in partial update #40272 " ( #40964 )
...
picks https://github.com/apache/doris/pull/40272
2024-09-26 22:54:27 +08:00
bf3d4240be
[fix](window_func) fix bug of agg function used in window function and add many test cases ( #40678 ) ( #41328 )
...
## Proposed changes
Issue Number: close #xxx
BP #40678
2024-09-26 22:50:34 +08:00
4deda2fce7
[improvement](nereids) Simplify ScanNode projection handling by removing redundant conditions ( #40801 ) ( #41315 )
...
pick from master #40801
This PR simplifies the handling of `ScanNode` projection logic.
Previously, the code included multiple conditional checks to determine
whether a `projectionTuple` should be generated. These conditions have
been removed, and now `projectionTuple `is always generated for
`ScanNode`, ensuring a consistent projection setup. Additionally,
redundant handling of `SlotId` and `SlotRef` has been eliminated, making
the code cleaner and easier to maintain. The behavior for `OlapScanNode`
remains unchanged.
2024-09-26 10:35:01 +08:00
a11fd62043
[fix](window function) Fix illegal frame range ( #41147 ) ( #41305 )
...
pick #41147
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# 0x00007F591D573520 in /lib/x86_64-linux-gnu/libc.so.6
2# pthread_kill at ./nptl/pthread_kill.c:89
3# raise at ../sysdeps/posix/raise.c:27
4# abort at ./stdlib/abort.c:81
5# _nl_load_domain at ./intl/loadmsgcat.c:1177
6# 0x00007F591D56AE96 in /lib/x86_64-linux-gnu/libc.so.6
7# doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false,
false, false, DefaultMemoryAllocator>, 16ul, 15ul>::operator[](long)
const at
/home/zcp/repo_center/doris_master/doris/be/src/vec/common/pod_array.h:365
8# doris::vectorized::ColumnNullable::is_null_at(unsigned long) const at
/home/zcp/repo_center/doris_master/doris/be/src/vec/columns/column_nullable.h:158
9#
doris::vectorized::ReaderFirstAndLastData<doris::vectorized::ColumnVector<double>,
true, true, false>::insert_result_into(doris::vectorized::IColumn&)
const at
/home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function_reader_first_last.h:125
10# doris::pipeline::AnalyticLocalState::_insert_result_info(long) in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
11# std::_Function_handler<void (long), std::_Bind_result<void, void
(doris::pipeline::AnalyticLocalState::*(doris::pipeline::AnalyticLocalState*,
std::_Placeholder<1>))(long)> >::_M_invoke(std::_Any_data const&,
long&&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
12# std::function<void (long)>::operator()(long) const at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560
13# doris::pipeline::AnalyticLocalState::_get_next_for_rows(unsigned
long) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 14#
std::enable_if<is_invocable_r_v<doris::Status, doris::Status
(doris::pipeline::AnalyticLocalState::*&)(unsigned long),
doris::pipeline::AnalyticLocalState*&, unsigned long>,
doris::Status>::type std::__invoke_r<doris::Status, doris::Status
(doris::pipeline::AnalyticLocalState::*&)(unsigned long),
doris::pipeline::AnalyticLocalState*&, unsigned long>(doris::Status
(doris::pipeline::AnalyticLocalState::*&)(unsigned long),
doris::pipeline::AnalyticLocalState*&, unsigned long&&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114
15# doris::Status std::_Bind_result<doris::Status, doris::Status
(doris::pipeline::AnalyticLocalState::*(doris::pipeline::AnalyticLocalState*,
std::_Placeholder<1>))(unsigned long)>::__call<doris::Status, unsigned
long&&, 0ul, 1ul>(std::tuple<unsigned long&&>&&, std::_Index_tuple<0ul,
1ul>) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:570
16# std::_Function_handler<doris::Status (unsigned long),
std::_Bind_result<doris::Status, doris::Status
(doris::pipeline::AnalyticLocalState::*(doris::pipeline::AnalyticLocalState*,
std::_Placeholder<1>))(unsigned long)> >::_M_invoke(std::_Any_data
const&, unsigned long&&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
17# std::function<doris::Status (unsigned long)>::operator()(unsigned
long) const at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560
18#
doris::pipeline::AnalyticSourceOperatorX::get_block(doris::RuntimeState*,
doris::vectorized::Block*, bool*) in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
19#
doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*,
doris::vectorized::Block*, bool*) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/operator.cpp:322
20# doris::pipeline::PipelineTask::execute(bool*) in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
21# doris::pipeline::TaskScheduler::_do_work(unsigned long) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/task_scheduler.cpp:138
22# doris::ThreadPool::dispatch_thread() in
/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
23# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 24#
start_thread at ./nptl/pthread_create.c:442
25# 0x00007F591D657850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
2024-09-26 09:55:33 +08:00
5b3b2cec80
[feat](metatable) support table$partitions for hive table ( #40774 ) ( #41230 )
...
bp #40774
and pick part of #34552 , add `isPartitionedTable()` interface in `TableIf`
2024-09-25 09:52:07 +08:00
8bb57bcc3e
[fix] (inverted index) fix the error in the query result when using count on index ( #41200 )
...
## Proposed changes
Introduced by #39473
<!--Describe your changes.-->
2024-09-24 19:47:18 +08:00
2b427c316a
[feature](functions) impl scalar functions normal_cdf,to_iso8601,from_iso8601_date ( #40695 ) ( #41049 )
...
bp #40695
2024-09-24 09:52:39 +08:00
0d38a9a36d
[feature](restore) support atomic restore ( #41107 )
...
Cherry-pick #40353 , #40734 , #40817 , #40876 , #40921 , #41017 , #41083
2024-09-24 09:41:41 +08:00
48e60f3ff3
[Fix](inverted index) fix wrong opt for count_on_index #41127 ( #41154 )
...
cherry pick from #41127
2024-09-23 22:45:52 +08:00
5bcea1983d
[opt](nereids) enable runtime filter use cte as target #40815 (2.1) ( #41090 )
...
## Proposed changes
pick #40815
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-23 22:34:03 +08:00
51bb4e9331
[fix](test-case) fix unstable test case about table_options sys table( #41006 ) ( #41121 )
...
bp #41006
2024-09-23 17:00:51 +08:00
f6917acd6a
[cherry-pick](branch2.1) Impl translate and url encode 2.1 ( #41051 )
...
## Proposed changes
pick https://github.com/apache/doris/pull/40567
some code about const folding should wait the pr picked:
https://github.com/apache/doris/pull/40441
2024-09-23 14:26:27 +08:00
e175c63d41
[Fix](inverted index) Fix wrong need read data opt when enable_common_expr_pushdown is disabled #40689 ( #41101 )
...
cherry pick from #40689
2024-09-23 14:21:30 +08:00
0e5c4281dc
[fix](function) fix Substring/SubReplace error result with input utf8… ( #40954 )
...
… string (#40929 )
https://github.com/apache/doris/pull/40929
```
mysql [(none)]>select sub_replace("你好世界","a",1);
+-------------------------------------+
| sub_replace('你好世界', 'a', 1) |
+-------------------------------------+
| �a�好世界 |
+-------------------------------------+
mysql [(none)]>select SUBSTRING('中文测试',5);
+------------------------------------------+
| substring('中文测试', 5, 2147483647) |
+------------------------------------------+
| 中文测试 |
+------------------------------------------+
1 row in set (0.04 sec)
now
mysql [(none)]>select sub_replace("你好世界","a",1);
+-------------------------------------+
| sub_replace('你好世界', 'a', 1) |
+-------------------------------------+
| 你a世界 |
+-------------------------------------+
1 row in set (0.05 sec)
mysql [(none)]>select SUBSTRING('中文测试',5);
+------------------------------------------+
| substring('中文测试', 5, 2147483647) |
+------------------------------------------+
| |
+------------------------------------------+
1 row in set (0.13 sec)
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-23 10:44:03 +08:00
7d64c8cbc6
[branch-2.1] Picks "[opt](autoinc) Remove some restrictions on schema change on table that has auto-increment column #40280 " ( #41096 )
...
picks https://github.com/apache/doris/pull/40280
2024-09-23 09:30:15 +08:00
9dc55f90eb
[opt](nereids) set lower bound for range-selectivity（2.1） ( #41061 )
...
## Proposed changes
pick #40089
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-22 07:32:22 +08:00
9877a08834
[feature](function) support ngram_search function #38226 ( #40893 )
...
https://github.com/apache/doris/pull/38226
mysql [test]>select ngram_search('123456789' , '12345' , 3);
+---------------------------------------+
| ngram_search('123456789', '12345', 3) |
+---------------------------------------+
| 0.6 |
+---------------------------------------+
1 row in set (0.01 sec)
mysql [test]>select ngram_search("abababab","babababa",2);
+-----------------------------------------+
| ngram_search('abababab', 'babababa', 2) |
+-----------------------------------------+
| 1 |
+-----------------------------------------+
1 row in set (0.01 sec)
```
doc https://github.com/apache/doris-website/pull/899
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-21 20:34:44 +08:00
d5115a21b5
[pick](ShortCircuit) Conjuncts outof key columns's order should be handled ( #41071 )
...
#37900
2024-09-21 20:34:05 +08:00
c744eb87c5
[fix](regression)fix some regression test ( #40928 ) ( #41046 )
...
bp #40928
2024-09-20 18:17:44 +08:00
1259fe2bd5
[fix](covar) Fix covar nullable on branch-2.1 ( #40841 )
...
covar should not be always nullable.
This fix on branch-2.1 makes covar same with master on FE.
2024-09-20 17:35:27 +08:00
8e860a26a7
[fix](systable) fix unstable case for partitions table ( #40553 ) ( #41043 )
...
bp #40553
2024-09-20 17:13:30 +08:00
6539c8fd35
[fix](decimal) throw overflow exception if result is NaN of Infinit when converting from decimal to float ( #40290 ) ( #41007 )
...
## Proposed changes
Issue Number: close #xxx
BP #40290
2024-09-20 14:05:51 +08:00
e0fac66223
[branch-2.1](fix) fix snappy decompressor bug ( #40862 )
...
## Proposed changes
Hadoop snappycodec source :
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/SnappyCodec.cc
Example:
OriginData(The original data will be divided into several large data
block.) :
large data block1 | large data block2 | large data block3 | ....
The large data block will be divided into several small data block.
Suppose a large data block is divided into three small blocks:
large data block1: | small block1 | small block2 | small block3 |
CompressData: <A [B1 compress(small block1) ] [B2 compress(small block1)
] [B3 compress(small block1)]>
A : original length of the current block of large data block.
sizeof(A) = 4 bytes.
A = length(small block1) + length(small block2) + length(small block3)
Bx : length of small data block bx.
sizeof(Bx) = 4 bytes.
Bx = length(compress(small blockx))
2024-09-20 11:57:14 +08:00
64880a10d6
[branch-2.1] Picks "[Fix](partial update) Fix partial update failed when merge_type=MERGE #40730 " ( #40951 )
...
picks https://github.com/apache/doris/pull/40730
2024-09-20 00:02:17 +08:00
5f583fa329
[branch-2.1][test](jdbc catalog) add oceanbase ce jdbc catalog test ( #40978 )
...
pick #34972 )
2024-09-19 22:11:24 +08:00
b8bc9b699c
[fix](scan) Incorrect scan keys lead to wrong query results. ( #40814 ) ( #40971 )
...
## Proposed changes
pick #40814
```
mysql [doris_14555]>select * from table_9436528_3;
+------+------+------+------+------------------------+--------------------+------+
| col1 | col2 | col3 | col5 | col4 | col6 | col7 |
+------+------+------+------+------------------------+--------------------+------+
| -100 | 1 | -82 | 1 | 2024-02-16 04:37:37.00 | -1299962421.904282 | NULL |
| -100 | 1 | 92 | 1 | 2024-02-16 04:37:37.00 | 23423423.0324234 | NULL |
| -100 | 0 | -82 | 0 | 2023-11-11 10:49:43.00 | 840968969.872149 | NULL |
```
wrong result:
```
mysql [doris_14555]>select * from table_9436528_3 where col1 <= -100 and col2 in (true, false) and col3 = -82;
+------+------+------+------+------------------------+--------------------+------+
| col1 | col2 | col3 | col5 | col4 | col6 | col7 |
+------+------+------+------+------------------------+--------------------+------+
| -100 | 1 | -82 | 1 | 2024-02-16 04:37:37.00 | -1299962421.904282 | NULL |
| -100 | 1 | 92 | 1 | 2024-02-16 04:37:37.00 | 23423423.0324234 | NULL |
+------+------+------+------+------------------------+--------------------+------+
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-19 22:01:02 +08:00
8302261dd2
[Fix](nereids) set all nullable aggregate function to alwaysnullable in window expression ( #40693 ) ( #40809 )
...
cherry-pick from master #40693
2024-09-19 15:19:06 +08:00
f483a7605c
[fix](nestedtypes) fix nested type with is_exclusive ( #40434 )
...
is_exclusive in column_array/map/struct has wrong semantic , we should
make sure it's nested column is also is_exclusive which can make
behavior right in some operator like join.
## Proposed changes
backport: https://github.com/apache/doris/pull/40398
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-19 12:28:51 +08:00
84f0b1fbfe
[feature](view) support create or replace view stmt ( #40715 ) ( #40915 )
...
pick #40715 to branch-2.1
2024-09-19 01:10:43 +08:00
774efe78e6
[fix](regression)fix maxcompute p0 case ( #40933 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-19 01:09:53 +08:00
4511d3e900
[cherry-pick](branch2.1) fix unstable case of partitions ( #40886 )
...
## Proposed changes
Issue Number: close #xxx
backport #40861
2024-09-18 09:49:24 +08:00
148f385901
[fix](tests) Fix export p2 2.1 ( #40852 )
...
bp: #40198
2024-09-15 21:38:43 +08:00
f3b1f1c19b
[fix](encrypt) wrong mode arg of encrypt and decrypt function make BE crash ( #40726 ) ( #40868 )
...
pick #40726 to branch-2.1
2024-09-15 21:31:00 +08:00
ea4d166edb
[branch-2.1] Picks "[opt](partial update) Allow to only specify key columns in partial update #40736 " ( #40863 )
...
picks https://github.com/apache/doris/pull/40736
2024-09-15 18:57:43 +08:00
cecd214345
[branch-2.1](Column) refactor ColumnNullable to provide flags safety ( #40769 ) ( #40848 )
...
pick https://github.com/apache/doris/pull/40769
Co-authored-by: Jerry Hu <mrhhsg@gmail.com >
2024-09-14 16:27:43 +08:00
