2e1ce758f1
[feature](function) support ip function ipv6numtostring(alias inet6_ntoa) ( #27342 )
2023-12-02 11:48:19 +08:00
baadc14e60
[Enhancement](function) support unix_timestamp with float ( #26827 )
...
---------
Co-authored-by: YangWithU <plzw8@outlook.com >
2023-11-27 09:58:53 +08:00
334260dff7
[feature](function) support ip function ipv4stringtonum(ordefault, ornull), inet_aton ( #25510 )
2023-11-17 10:27:07 +08:00
2c6d2255c3
[fix](Nereids) nested type literal type coercion and insert values with map ( #26669 )
2023-11-14 21:13:26 -06:00
9a4fd5be79
[nereids](datetime) fix wrong result type of datetime add with interval as first arg ( #26957 )
...
Incorrect result data type cause be cordump:
drop table if exists testaaa;
create table testaaa(k1 tinyint, k2 smallint, k3 int, k4 bigint, k5 decimal(9,3), k6 char(5), k10 date, k11 datetime, k7 varchar(20), k8 double max, k9 float sum) engine=olap distributed by hash(k1) buckets 5 properties("storage_type"="column","replication_num"="1") ;
insert into testaaa values(1,1,1,1,9.3, "k6", "2023-11-14", "2023-11-14", "k7", 9.99, 9.99);
select interval 10 year + k10 from testaaa;
The plan result type is DATE:
mysql [test]>explain verbose select interval 10 year + k10 from testaaa;
+-------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+-------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| years_add(k10, INTERVAL 10 YEAR)[#11 ] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCAL |
| |
| 64:VEXCHANGE |
| offset: 0 |
| tuple ids: 1N |
| |
| PLAN FRAGMENT 1 |
| |
| PARTITION: HASH_PARTITIONED: k1[#0 ] |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| STREAM DATA SINK |
| EXCHANGE ID: 64 |
| UNPARTITIONED |
| |
| 58:VOlapScanNode |
| TABLE: default_cluster:test.testaaa(testaaa), PREAGGREGATION: OFF. Reason: No aggregate on scan. |
| partitions=1/1 (testaaa), tablets=5/5, tabletList=945025,945027,945029 ... |
| cardinality=1, avgRowSize=9885.0, numNodes=1 |
| pushAggOp=NONE |
| projections: years_add(k10[#6 ], INTERVAL 10 YEAR) |
| project output tuple id: 1 |
| tuple ids: 0 |
| |
| Tuples: |
| TupleDescriptor{id=0, tbl=testaaa, byteSize=8} |
| SlotDescriptor{id=6, col=k10, colUniqueId=6, type=DATEV2, nullable=true, isAutoIncrement=false} |
| |
| TupleDescriptor{id=1, tbl=testaaa, byteSize=32} |
| SlotDescriptor{id=11, col=null, colUniqueId=null, type=DATE, nullable=true, isAutoIncrement=false} |
+-------------------------------------------------------------------------------------------------------+
39 rows in set (1 min 31.50 sec)
coredump stack:
F1109 20:11:37.677680 323805 assert_cast.h:61] Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnVector
*** Check failure stack trace: ***
F1109 20:11:37.680608 323800 assert_cast.h:61] Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnVector
*** Check failure stack trace: ***
F1109 20:11:37.680608 323800 assert_cast.h:61] Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnVectorF1109 20:11:37.681102 323808 assert_cast.h:61] Bad cast from type:doris::vectorized::ColumnVector to doris::vectorized::ColumnVector
*** Check failure stack trace: ***
@ 0x56489d591d3d google::LogMessage::Fail()
@ 0x56489d591d3d google::LogMessage::Fail()
@ 0x56489d591d3d google::LogMessage::Fail()
@ 0x56489d594279 google::LogMessage::SendToLog()
@ 0x56489d594279 google::LogMessage::SendToLog()
@ 0x56489d594279 google::LogMessage::SendToLog()
@ 0x56489d5918a6 google::LogMessage::Flush()
@ 0x56489d5918a6 google::LogMessage::Flush()
@ 0x56489d5918a6 google::LogMessage::Flush()
@ 0x56489d5948e9 google::LogMessageFatal::~LogMessageFatal()
@ 0x56489d5948e9 google::LogMessageFatal::~LogMessageFatal()
@ 0x56489d5948e9 google::LogMessageFatal::~LogMessageFatal()
@ 0x56487a2a8a0c assert_cast<>()
@ 0x56487a2a8a0c assert_cast<>()
@ 0x56487a2a8a0c assert_cast<>()
@ 0x5648893d8312 doris::vectorized::ColumnVector<>::insert_range_from()
@ 0x5648893d8312 doris::vectorized::ColumnVector<>::insert_range_from()
@ 0x5648893d8312 doris::vectorized::ColumnVector<>::insert_range_from()
@ 0x56488924a670 doris::vectorized::ColumnNullable::insert_range_from()
@ 0x56488924a670 doris::vectorized::ColumnNullable::insert_range_from()
@ 0x56488924a670 doris::vectorized::ColumnNullable::insert_range_from()
@ 0x56487a454475 doris::ExecNode::do_projections()
@ 0x56487a454475 doris::ExecNode::do_projections()
@ 0x56487a454475 doris::ExecNode::do_projections()
@ 0x56487a454b89 doris::ExecNode::get_next_after_projects()
@ 0x56487a454b89 doris::ExecNode::get_next_after_projects()
*** Query id: a467995b35334741-b625042f56495aaf ***
*** tablet id: 0 ***
*** Aborted at 1699531898 (unix time) try "date -d @1699531898" if you are using GNU date ***
*** Current BE git commitID: 0d83327a7c ***
*** SIGABRT unknown detail explain (@0x190d64) received by PID 1641828 (TID 1642168 OR 0x7f6ff96c0700) from PID 1641828; stack trace: ***
@ 0x556ca2a3ab8f std::_Function_handler<>::_M_invoke()
@ 0x556c9f322787 std::function<>::operator()()
@ 0x556ca29da0b0 doris::Thread::supervise_thread()
@ 0x556c9f322787 std::function<>::operator()()
@ 0x7f71b9c38609 start_thread
@ 0x556ca29da0b0 doris::Thread::supervise_thread()
@ 0x7f71b9c38609 start_thread
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/signal_handler.h:417
1# 0x00007F71B9E09090 in /lib/x86_64-linux-gnu/libc.so.6
2# raise at ../sysdeps/unix/sysv/linux/raise.c:51
3# abort at /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81
4# 0x0000556CC51F3729 in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
5# 0x0000556CC51E8D3D in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
6# google::LogMessage::SendToLog() in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
7# google::LogMessage::Flush() in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
8# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
9# doris::vectorized::ColumnVector const& assert_cast const&, doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&) in /mnt/hdd01/ci/branch20-deploy/be/lib/doris_be
10# doris::vectorized::ColumnVector::insert_range_from(doris::vectorized::IColumn const&, unsigned long, unsigned long) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/vec/columns/column_vector.cpp:354
11# doris::vectorized::ColumnNullable::insert_range_from(doris::vectorized::IColumn const&, unsigned long, unsigned long) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/vec/columns/column_nullable.cpp:289
12# doris::ExecNode::do_projections(doris::vectorized::Block*, doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/exec/exec_node.cpp:573
13# doris::ExecNode::get_next_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*, std::function const&, bool) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/exec/exec_node.cpp:592
14# doris::pipeline::SourceOperator::get_block(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState&) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/pipeline/exec/operator.h:413
15# doris::pipeline::PipelineTask::execute(bool*) at /home/zcp/repo_center/doris_branch-2.0/doris/be/src/pipeline/pipeline_task.cpp:259
16# doris::pipeline::TaskScheduler::_do_work(unsigned
2023-11-14 20:28:41 +08:00
fd92810df8
[FIX](regresstest)fixed out file for test-cast-map-function test ( #26389 )
2023-11-03 05:47:10 -05:00
93a934e775
[Improve](map) support map cast with map literal and implicate nested scala cast ( #26126 )
2023-11-02 09:56:42 -05:00
c46fa339d0
[feature](Nereids): make count() avaliable for all data type ( #26180 )
...
make count() avaliable for all data type
2023-11-02 17:20:24 +08:00
3e10e5af39
[Fix](Serde) Fix content displayed by complex types in MySQL Client ( #25946 )
...
This pr makes three changes to the display of complex types：
1. NULL value in complex types refers to being displayed as `null`, not `NULL`
2. struct type is displayed as "column_name": column_value
3. Time types such as `datetime` and `date`, are displayed with double quotes in complex types. like
`{1, "2023-10-26 12:12:12"}`
This pr also do a code refactor:
1. nesting_level is set to a member variable of the `DataTypeSerDe`, rather than a parameter in methods.
What's more, this pr fix a bug that fileSize is not correct, introduced by this pr: #25854
2023-11-01 23:48:55 +08:00
3a954cd1aa
[fix](function)return NULL rather than 'null' if path not found ( #25880 )
...
fix json_extract not return NULL but null
2023-10-30 14:26:44 +08:00
57340a4187
[feature](nereids) add function array_agg ( #25630 )
2023-10-23 02:05:54 -05:00
e77b98be88
[fix](months_diff) fix wrong result of months_diff ( #25577 )
2023-10-19 14:29:47 +08:00
2ddd2e5079
[feature](Nereids) add map_agg function ( #25246 )
2023-10-18 06:44:36 -05:00
62d06584f1
[feature](fe) add function 'BitmapAgg' in nereids ( #25508 )
2023-10-18 14:24:27 +08:00
af8832389f
[feature](Nereids) add 4 array functions ( #25488 )
...
- array_concat
- array_pushback
- array_pushfront
- array_zip
2023-10-17 04:45:15 -05:00
4c57c31c5c
[fix](Nereids) count should not accept complex and json type ( #25354 )
2023-10-15 22:08:35 -05:00
42f8b253aa
[function](nereids) support array_apply/array_repeat/group_uniq_array/ipv4numtostring ( #25249 )
...
nereids support functions: array_apply/array_repeat/group_uniq_array/ipv4numtostring
2023-10-12 11:08:42 +08:00
b91bce8a62
[feature](Nereids) add array distance functions ( #25196 )
...
- l1_distance
- l2_distance
- cosine_distance
- inner_product
2023-10-10 21:35:06 -05:00
181c58c691
[fix](Nereids) count_by_enum signature is wrong ( #25167 )
2023-10-10 13:05:20 +08:00
37247ac449
[opt](Nereids) add two args signature to trim family functions ( #25169 )
2023-10-09 07:17:52 -05:00
320709b9ff
[opt](Nereids) support like and regexp function ( #25148 )
2023-10-09 02:55:57 -05:00
3a45001447
[fix](Nerids) fix error when the view has lambda functions ( #25067 )
...
1. To ensure compatibility with the original optimizer, expose the non-lambda signature of highorder function externally.
2. fix some bugs in toSql function in the original optimizer
2023-10-08 15:45:24 +08:00
12bee54cf7
[fix](Nereids) function implict cast for json is not work well ( #24852 )
...
for sql like:
SELECT JSONB_EXTRACT('{"k1":"v31","k2":300}','$.k1');
the result should be
+------------------------------------------------+
| jsonb_extract('{"k1":"v31","k2":300}', '$.k1') |
+------------------------------------------------+
| "v31" |
+------------------------------------------------+
but curent result is
+------------------------------------------------+
| jsonb_extract('{"k1":"v31","k2":300}', '$.k1') |
+------------------------------------------------+
| <null> |
+------------------------------------------------+
2023-09-25 21:10:04 +08:00
22616d125d
[function](bitmap) add function alias bitmap_andnot and bitmap_andnot_count ( #24771 )
2023-09-22 12:18:31 +08:00
5b590bbfcf
[feat](Nereids) add lambda func array_last and array_first ( #24682 )
2023-09-21 12:23:34 +08:00
a3361df7b9
[Feat](Nereids) support json and jsonb datatype ( #24156 )
...
Feature:
support jsonb and json type in nereids
Document:
this feature supports these two datatype in nereids optimizer like original planner, the sql reference is same as before
[JSON - Apache Doris](https://doris.apache.org/zh-CN/docs/dev/sql-manual/sql-reference/Data-Types/JSON )
2023-09-20 14:32:22 +08:00
e9435c14f8
[Improve](array-func)improve array union support multi params ( #24327 )
2023-09-20 14:29:48 +08:00
c3bd2a22d4
[feature](Nereids) add many array functions ( #24301 )
...
Add function array_filter, array_sortby, array_last_index, array_first_index, array_orderby, array_count
2023-09-19 18:58:49 +08:00
4b5cea1ef8
[enhancement](fix)change ordinary type null value is \N,complex type null value is null ( #24207 )
2023-09-16 21:46:42 +08:00
88adab3114
[fix](Nereids): fix be core when array_map is not nullable ( #24488 )
...
fix be core when array_map is not nullable
2023-09-16 20:39:15 +08:00
4fbb25bc55
[Enhancement](function) Support date_trunc(date) and use it in auto partition ( #24341 )
...
Support date_trunc(date) and use it in auto partition
2023-09-14 16:53:09 +08:00
268c867679
[Improve](serde)replace function_cast from_string to serde ( #24087 )
...
Now we can not support streamload with column which is map/array nested map/array
serde can do this now , so we can replace it
Notice. if item data in complex type data is empty we just return error, instead of makeup default value , because now we can not define right default for complex type
2023-09-14 13:53:16 +08:00
dc28878f0e
[FIX](function) fix size function for array map ( #23920 )
...
Issue Number: close #xxx
now we use select size(map(1, 2)); which will make be core
and we can make size function handle array & map column both
2023-09-06 14:32:06 +08:00
a136836770
[feature](Nereids) add two functions: char and covert ( #23104 )
...
add [char](https://doris.apache.org/zh-CN/docs/dev/sql-manual/sql-functions/string-functions/char/?_highlight=char ) func
```
mysql> select char(68, 111, 114, 105, 115);
+--------------------------------------+
| char('utf8', 68, 111, 114, 105, 115) |
+--------------------------------------+
| Doris |
+--------------------------------------+
```
convert func
```
MySQL root@127.0.0.1:(none)> select convert(1 using gbk);
+-------------------+
| convert(1, 'gbk') |
+-------------------+
| 1 |
+-------------------+
```
2023-08-30 17:09:06 +08:00
f7d2c1faf6
[feature](Nereids) support select key encryptKey ( #23257 )
...
Add select key
```
- CREATE ENCRYPTKEY key_name AS "key_string"
- select key my_key
+-----------------------------+
| encryptKeyRef('', 'my_key') |
+-----------------------------+
| ABCD123456789 |
+-----------------------------+
```
2023-08-28 14:07:26 +08:00
e84989fb6d
[feature](Nereids) support map type ( #23493 )
2023-08-28 11:31:44 +08:00
0ccb7262a7
[feature](Nereids) add password func ( #23244 )
...
add password function
```
select password("123");
+-------------------------------------------+
| password('123') |
+-------------------------------------------+
| *23AE809DDACAF96AF0FD78ED04B6A265E05AA257 |
+-------------------------------------------+
```
2023-08-25 14:04:49 +08:00
3786ffec51
[opt](Nereids) add some array functions ( #23324 )
...
1. rename TVFProperties to Properties
2. add generating function explode and explode_outer
3. fix concat_ws could not apply on array
4. check tokenize second argument format on FE
5. add test case for concat_ws, tokenize, explode, explode_outer and split_by_string
2023-08-25 11:01:50 +08:00
daa4db097e
[fix](Nereids) array_difference and array_position get wrong result ( #23331 )
...
1. change array_difference signature to let it return same type as arg
2. do not change precision when signature not use wildcard type
2023-08-23 20:38:09 +08:00
51ac92f65c
Revert "[fix](function) to_bitmap parameter parsing failure returns null instead of bitmap_empty ( #21236 )" ( #23368 )
...
This reverts commit 1c3cc77a54938ed948ad8186b8dea8385977d23c.
2023-08-23 18:27:35 +08:00
da2eb69eba
[test](Nereids) add array scalar function test cases ( #23303 )
2023-08-22 15:05:28 +08:00
1c3cc77a54
[fix](function) to_bitmap parameter parsing failure returns null instead of bitmap_empty ( #21236 )
...
* [fix](function) to_bitmap parameter parsing failure returns null instead of bitmap_empty
* add ut
* fix nereids
* fix regression-test
2023-08-18 14:37:49 +08:00
d3b50e3b2a
[BUG](date_trunc) fix date_trunc function only handle lower string ( #22602 )
...
fix date_trunc function only handle lower string
2023-08-05 12:53:13 +08:00
ae5e39ad26
[opt](Nereids) add double signature back for round like function ( #22284 )
...
add double signature back for round like function
2023-07-27 19:10:43 +08:00
fb41265c27
[opt](Nereids) add boolean type signature for sum aggregate function ( #21959 )
2023-07-27 09:41:19 +08:00
21deb57a4d
[fix](Nereids) remove double sigature of ceil, floor and round ( #22134 )
...
we convert input parameters to double for function ceil, floor and round,
because DecimalV2 could not do these operation. Since we intro DecimalV3,
we should convert all parameters to DecimalV3 to get correct result.
For example, when we use double as parameters, we get wrong result:
```sql
select round(341/20000,4),341/20000,round(0.01705,4);
+-------------------------+---------------+-------------------+
| round((341 / 20000), 4) | (341 / 20000) | round(0.01705, 4) |
+-------------------------+---------------+-------------------+
| 0.017 | 0.01705 | 0.0171 |
+-------------------------+---------------+-------------------+
```
DecimalV3 could get correct result
```sql
select round(341/20000,4),341/20000,round(0.01705,4);
+-------------------------+---------------+-------------------+
| round((341 / 20000), 4) | (341 / 20000) | round(0.01705, 4) |
+-------------------------+---------------+-------------------+
| 0.0171 | 0.01705 | 0.0171 |
+-------------------------+---------------+-------------------+
```
2023-07-24 16:08:00 +08:00
5fe7106b83
[Bug](planner) fix pre condition check fail on max(null) ( #20509 )
...
fix pre condition check fail on max(null)
2023-06-08 14:49:52 +08:00
1d05feea1b
[Feature](Nereids) add executable function to support fold constant for functions ( #18209 )
...
1. Add date-time functions for fold constant for Nereids.
This is the list of executable date-time function nereids supports up to now:
- now()
- now(int)
- current_timestamp()
- current_timestamp(int)
- localtime()
- localtimestamp()
- curdate()
- current_date()
- curtime()
- current_time()
- date_{add/sub}(),{years/months/days/hours/minutes/seconds}_{add/sub}()
- datediff()
- {date/datev2}()
- {year/quarter/month/day/hour/minute/second}()
- dayof{year/month/week}()
- date_format()
- date_trunc()
- from_days()
- last_day()
- to_monday()
- from_unixtime()
- unix_timestamp()
- utc_timestamp()
- to_date()
- to_days()
- str_to_date()
- makedate()
2. solved problem:
- enable datev2/datetimev2 default.
- refactor Nereids foldConstantOnFE and support fold nested expression.
- separate the executable into multi-files for easily-reading and adding new functions
2023-05-17 21:26:31 +08:00
ec1ab1a3d2
[Improve](GEO)wkb input and output are represented as hexadecimal strings And delete EWKB ( #18721 )
2023-04-21 15:11:18 +08:00
f2d75cb492
[fix](Nereids) fix signature precision round for decimalv3 ( #18639 )
...
add decimalv3 signature to below functions:
ceil
dceil
dfloor
dround
floor
round
round_bankers
truncate
fix ComputePrecisionForRound to get correct signature
2023-04-14 18:18:41 +08:00
