a45dc8796a
[fix](Nereids) simplify decimal comparison wrong when cast to smaller scale ( #41151 ) ( #41618 )
...
pick from master #41151
2024-10-09 23:03:01 +08:00
2b427c316a
[feature](functions) impl scalar functions normal_cdf,to_iso8601,from_iso8601_date ( #40695 ) ( #41049 )
...
bp #40695
2024-09-24 09:52:39 +08:00
f6917acd6a
[cherry-pick](branch2.1) Impl translate and url encode 2.1 ( #41051 )
...
## Proposed changes
pick https://github.com/apache/doris/pull/40567
some code about const folding should wait the pr picked:
https://github.com/apache/doris/pull/40441
2024-09-23 14:26:27 +08:00
9877a08834
[feature](function) support ngram_search function #38226 ( #40893 )
...
https://github.com/apache/doris/pull/38226
mysql [test]>select ngram_search('123456789' , '12345' , 3);
+---------------------------------------+
| ngram_search('123456789', '12345', 3) |
+---------------------------------------+
| 0.6 |
+---------------------------------------+
1 row in set (0.01 sec)
mysql [test]>select ngram_search("abababab","babababa",2);
+-----------------------------------------+
| ngram_search('abababab', 'babababa', 2) |
+-----------------------------------------+
| 1 |
+-----------------------------------------+
1 row in set (0.01 sec)
```
doc https://github.com/apache/doris-website/pull/899
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-21 20:34:44 +08:00
f3b1f1c19b
[fix](encrypt) wrong mode arg of encrypt and decrypt function make BE crash ( #40726 ) ( #40868 )
...
pick #40726 to branch-2.1
2024-09-15 21:31:00 +08:00
963415ce45
[test](case) add some test case for encrypt/decrypt functions ( #40427 ) ( #40847 )
...
pick #40427 to branch-2.1
Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com >
2024-09-14 16:26:39 +08:00
9a79edca84
[cherry-pick](branch-21) fix partition_topn not reset output rows after do_partition_topn_sort ( #40761 ) ( #40792 )
...
## Proposed changes
cherry-pick from master https://github.com/apache/doris/pull/40761
<!--Describe your changes.-->
2024-09-14 11:15:56 +08:00
3246baa451
[branch-2.1](function) Refine crypto functions signature to fix wrong result( #40285 ) ( #40648 )
...
pick https://github.com/apache/doris/pull/40285
2024-09-11 15:32:19 +08:00
0928c9c6ed
[fix](unary function) Fix wrong result of asin, acos and sqrt when processing invalid input #40267 ( #40358 )
...
cherry pick from #40267
2024-09-05 19:51:01 +08:00
70eeda9f50
[cherry-pick](branch-21) fix encryption_function return wrong result at const column ( #40201 ) ( #40247 )
...
## Proposed changes
cherry-pick from master #40201
the string object have error of order, should be src, key,iv,mode
2024-09-02 16:09:15 +08:00
163193b1d4
[branch-2.1](function) fix random_bytes return same data for multi rows ( #39891 ) ( #40137 )
...
pick https://github.com/apache/doris/pull/39891
Issue Number: close #xxx
before:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a |
+------------------------+
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
+------------------------+
```
now:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a |
+------------------------+
| 0xd82cf60825b29ef2a0fd |
| 0x6f8c808415bdbaa6d257 |
| 0x7c26b5214297a151c25c |
| 0x43f02c77293063900437 |
| 0x5e5727569dec5e24f96b |
| 0x434f20bf74d7759640b7 |
| 0x087ed96b739750c733a6 |
| 0xdf05f6d7ede4972eb846 |
| 0xcefab471912264b5c54f |
| 0x1bddc019409d1926aa10 |
+------------------------+
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-30 10:43:42 +08:00
b052f46944
[fix](json-quote) fix json quote func for not find the func ( #39931 ) ( #40061 )
...
1. fix function not found before this pr:
```
drop table if exists t003;
create table t003 (a bigint, b json not null) properties ("replication_num"="1");
insert into t003 values (1, '{"a":1,"b":2}');
select a, map_agg("k1", json_quote(b)) from t003 group by a;
[17:47]> select a, map_agg("k1", json_quote(b)) from t003 group by a;
(1105, 'errCode = 2, detailMessage = (172.20.48.119)[INTERNAL_ERROR]Function json_quote get failed, expr is VectorizedFnCall[json_quote](arguments=(CAST b(JSONB) TO String),return=Nullable(String)) and return type is Nullable(String).')
```
after pr , we can carry on it
2. fix a core if we use json_quote with table column
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
*** Query id: b7b94dead55e4090-8c8e94f65ec9efd3 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1724744881 (unix time) try "date -d @1724744881" if you are using GNU date ***
*** Current BE git commitID: bd5844ea0d ***
*** SIGABRT unknown detail explain (@0x461003611e1) received by PID 3543521 (TID 3548858 OR 0x7f8a53c1a700) from PID 3543521; stack trace: ***
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
1# 0x00007F9429595B50 in /lib64/libc.so.6
2# gsignal in /lib64/libc.so.6
3# __GI_abort in /lib64/libc.so.6
4# _nl_load_domain.cold.0 in /lib64/libc.so.6
5# 0x00007F942958E426 in /lib64/libc.so.6
6# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::Prefix(rapidjson::Type) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488
7# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::String(char const*, unsigned int, bool) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:206
8# bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u> >(rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>&) const at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/document.h:1974
9# doris::vectorized::FunctionJsonQuoteImpl::execute(std::vector<doris::vectorized::ColumnStr<unsigned int> const*, std::allocator<doris::vectorized::ColumnStr<unsigned int> const*> > const&, doris::vectorized::ColumnStr<unsigned int>&, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:826
10# doris::vectorized::FunctionJson<doris::vectorized::FunctionJsonQuoteImpl>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:985
11# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:463
12# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:120
13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:245
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-28 21:31:08 +08:00
4cf769b39f
[Improve](table-function) explode json array with json args ( #39491 )
2024-08-27 14:53:17 +08:00
21bd4a4ac8
[bug](function)fix json_replace check return type error ( #37014 ) ( #39938 )
...
1. fix the return type dcheck error:
```
mysql [test]>select (json_replace(a, '$.fparam.nested_2', "qwe")) from json_table_2 limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INTERNAL_ERROR]Function json_replace get failed, expr is VectorizedFnCall[json_replace](arguments=a, String, String, String,return=Nullable(String)) and return type is Nullable(String).
```
2. improve the json_replace/json_insert/json_set function execute of not
convert const column, test about could faster 1s on 1000w table rows
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-27 08:03:48 +08:00
f7b6ed772b
[Fix-2.1](testcase) Fix some testcases unstable results due to lack of orderby ( #39892 )
...
## Proposed changes
Issue Number: close #xxx
some sql's order-by column has same value, add more order-by column to
fix it.
2024-08-25 08:45:36 +08:00
bb687bd69c
[cherry-pick](branch-2.1) add function regexp_extract_or_null ( #39561 )
...
# Proposed changes
pick https://github.com/apache/doris/pull/38296
2024-08-21 09:14:58 +08:00
7687f2c53a
[fix](ip-funcs) fix ip inet6_aton funcs #39415 ( #39513 )
2024-08-17 10:56:06 +08:00
226e01889c
[fix](array_apply) pick array apply fix ( #39328 )
...
## Proposed changes
backport: https://github.com/apache/doris/pull/39105
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-14 18:52:29 +08:00
5f77f909d9
[cherry-pick](branch-2.1) Pick "[feature](function) support ip functions named ipv4_to_ipv6 and cut_ipv6" ( #39058 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
pick https://github.com/apache/doris/pull/36883 and
https://github.com/apache/doris/pull/35239
2024-08-10 18:37:11 +08:00
c7b59b38ef
[fix](hist) Fix unstable result of aggregrate function hist #38608 ( #38893 )
...
cherry pick from #38608
2024-08-06 08:52:03 +08:00
e9052e2180
[cherry-pick](branch-21) fix mod function cause core dump ( #37999 ) ( #38308 )
...
## Proposed changes
cherry-pick from master https://github.com/apache/doris/pull/37999
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-25 12:06:21 +08:00
79a6496bb6
[branch-2.1](function) fix wrong result when convert_tz is out of bound ( #37358 ) ( #38313 )
...
## Proposed changes
pick https://github.com/apache/doris/pull/37358
before:
```sql
mysql> select CONVERT_TZ(cast('0000-01-01 00:00:00.00001' as DATETIMEV1), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533)));
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| convert_tz(cast('0000-01-01 00:00:00.00001' as DATETIME), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533))) |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| q535-12-31 08:01:19 |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.12 sec)
```
now:
```sql
mysql> select CONVERT_TZ(cast('0000-01-01 00:00:00.00001' as DATETIMEV1), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533)));
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| convert_tz(cast('0000-01-01 00:00:00.00001' as DATETIME), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533))) |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| NULL |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.09 sec)
```
2024-07-25 11:32:44 +08:00
8de13c5cc8
[fix](function) error scale set in unix_timestamp ( #36110 ) ( #37619 )
...
## Proposed changes
```
mysql [test]>set DEBUG_SKIP_FOLD_CONSTANT = true;
Query OK, 0 rows affected (0.00 sec)
mysql [test]>select cast(unix_timestamp("2024-01-01",'yyyy-MM-dd') as bigint);
+------------------------------------------------------------+
| cast(unix_timestamp('2024-01-01', 'yyyy-MM-dd') as BIGINT) |
+------------------------------------------------------------+
| 1704038400000000 |
+------------------------------------------------------------+
```
now
```
mysql [test]>select cast(unix_timestamp("2024-01-01",'yyyy-MM-dd') as bigint);
+------------------------------------------------------------+
| cast(unix_timestamp('2024-01-01', 'yyyy-MM-dd') as BIGINT) |
+------------------------------------------------------------+
| 1704038400 |
+------------------------------------------------------------+
1 row in set (0.01 sec)
```
The column does not have a scale set, but the cast uses the scale to
perform the cast.
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-15 10:00:04 +08:00
ca0e44f83f
[fix](case) fix struct format out files ( #37350 ) ( #37499 )
...
bp #37350
2024-07-09 10:11:50 +08:00
ceef9ee123
[feature](serde) support presto compatible output format ( #37039 ) ( #37253 )
...
bp #37039
2024-07-04 13:56:05 +08:00
d237a4d303
[fix](array)fix array_except/union for left const return only one row result #36776 ( #36986 )
2024-06-30 12:25:17 +08:00
cbaff8a700
[fix](nereids)change the decimal's precision and scale for cast(xx as decimal) ( #36540 )
...
pick from master #36316
expression cast( xx as decimal )'s datatype maybe decimalv3 or decimalv2
depending on enable_decimal_conversion value in fe conf file. if
enable_decimal_conversion is true, the datatype is decimalv3(9, 0), but
the datatype was decimalv3(38, 9) in 2.0 releases. So this pr change the
datatype same as 2.0 releases to keep the behavior consistent.
2024-06-20 17:46:11 +08:00
4a117800ca
[Bug](Function) fix json contains with empty value ( #36320 ) ( #36418 )
2024-06-18 10:20:45 +08:00
dd567fa774
[fix](function) support return JsonType for If function ( #35199 )
...
add a FunctionSignature for If to support return Type is JsonType.
2024-05-24 16:23:58 +08:00
eb7eaee386
[fix](function) money format ( #34680 )
2024-05-18 18:35:29 +08:00
ca9eb56233
[Fix](functions) fix strcmp return value #34565
2024-05-12 09:49:38 +08:00
58c19e33b3
[fix](round) Fix incorrect decimal scale inference in round functions ( #34471 )
...
* FIX NEEDED
* FORMAT
* FORMAT
* FIX TEST
2024-05-11 11:42:12 +08:00
e38801968d
[Fix](functions) Fix bug in makedate and str_to_date functions
2024-05-10 22:14:25 +08:00
845732b440
[WIP](test) remove enable_nereids_planner in regression cases (part 3) ( #34558 )
...
previous PR:
part 1: #34417
part 2: #34490
2024-05-10 22:11:01 +08:00
9b712b03b4
[FIX]fix is_ip_address_in_range func with const param ( #34266 )
2024-05-10 14:37:20 +08:00
f7900b53ce
[enhancement](function) floor/ceil/round/round_bankers can use column as scale argument ( #34391 )
2024-05-06 22:18:36 +08:00
20bd0c2987
[FIX](cases )fix ipv6 value for regress case
2024-04-29 13:37:29 +08:00
8d98c71079
[FIX]fix cidr func with const param ( #33968 )
2024-04-24 17:13:50 +08:00
2f60dcf890
[test](hll) fix unstable case without order by clause ( #33947 )
2024-04-24 17:13:50 +08:00
22a6b1d3f5
[feature](function) support hll functions hll_from_base64, hll_to_base64 ( #32089 )
...
Issue Number: #31320
Support two hll functions:
- hll_from_base64
Convert a base64 string(result of function hll_to_base64) into a hll.
- hll_to_base64
Convert an input hll to a base64 string.
2024-04-17 23:42:13 +08:00
bf022f9d8d
[enhancement](function truncate) truncate can use column as scale argument ( #32746 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-10 14:53:56 +08:00
1f1932c6b7
[enhancement](nereids)add some date functions for constant fold ( #32772 )
2024-04-10 11:34:30 +08:00
2a0644f442
[Fix](function) Fix unix_timestamp core for string input ( #32871 )
2024-04-09 12:48:35 +08:00
ccd21a6ea4
[Improve](InPredict) enhance in predict with array type ( #31828 )
2024-03-12 14:19:14 +08:00
e91d16854b
[fix](function) fix date_format function execution error on fe ( #31645 )
2024-03-07 16:53:19 +08:00
2d6e975d5a
[fix](cast) fix wrong result while cast string to float ( #31781 )
...
Issue Number: close #31518
2024-03-06 13:07:59 +08:00
7998da4691
[fix](cast) wrong result while cast const to double then to string ( #31657 )
...
Issue Number: close #31514
2024-03-06 13:06:27 +08:00
cca0773261
[feature](function) round function defaults to rounding normally
2024-03-06 13:06:26 +08:00
ff3c7765d6
[Bug](Func) Fix negative number in bitmap return error result ( #31563 )
2024-02-29 12:38:03 +08:00
f163d56a98
[feature](function) support sequence function(alias of array_range), enhance both to handle datetimev2 ( #30823 )
2024-02-27 10:12:19 +08:00