database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
4,295 Commits 1 Branch 0 Tags
8a066e2586a0ffc66feb0f3c2ed4c705f4c607f8
Commit Graph

54 Commits

Author SHA1 Message Date
Pxl
8a066e2586 [fix](vectorized) core dump on ST_AsText (#8870) 2022-04-11 09:39:32 +08:00
Gabriel
7f7172807f [feature](function)(vectorized) Support all geolocation functions on vectorized engine (#8846) 2022-04-11 09:36:53 +08:00
Gabriel
0d761f9909 [feature-wip][UDF][DIP-1] Support variable-size input and output for Java UDF (#8678) 
This feature is proposed in DSIP-1. This PR support variable-length input and output Java UDF.
 2022-04-11 09:36:16 +08:00
zbtzbtzbt
6ed59bb98b [refactor](code_style) remove useless inline #8933 
1.Member functions defined in a class are inline by default (implicitly), and do not need to be added
2.inline is a keyword used for implementation, which has no effect when placed before the function declaration
 2022-04-10 18:29:55 +08:00
camby
c5718928df [feature-wip](array-type) support explode and explode_outer table function (#8766) 
explode(ArrayColumn) desc:
> Create a row for each element in the array column. 

explode_outer(ArrayColumn) desc:
> Create a row for each element in the array column. Unlike explode, if the array is null or empty, it returns null.

Usage example:
1. create a table with array column, and insert some data;
2. open enable_lateral_view and enable_vectorized_engine;
```
set enable_lateral_view = true;
set enable_vectorized_engine=true;
```
3. use explode_outer
```
> select * from array_test;
+------+------+--------+
| k1   | k2   | k3     |
+------+------+--------+
|    3 | NULL | NULL   |
|    1 |    2 | [1, 2] |
|    2 |    3 | NULL   |
|    4 | NULL | []     |
+------+------+--------+

> select k1,explode_column from array_test LATERAL VIEW explode_outer(k3) TempExplodeView as explode_column;
+------+----------------+
| k1   | explode_column |
+------+----------------+
|    1 |              1 |
|    1 |              2 |
|    2 |           NULL |
|    4 |           NULL |
|    3 |           NULL |
+------+----------------+
```
4. explode usage example. explode return empty rows while the ARRAY is null or empty
```
> select k1,explode_column from array_test LATERAL VIEW explode(k3) TempExplodeView as explode_column;
+------+----------------+
| k1   | explode_column |
+------+----------------+
|    1 |              1 |
|    1 |              2 |
+------+----------------+
```
 2022-04-08 12:11:04 +08:00
Pxl
dbbc6549bd [feature](vectorized) support vexplode_bitmap (#8890) 2022-04-08 09:20:26 +08:00
Gabriel
3f04220d49 [typo] Fix typo in function.cpp (#8873) 2022-04-08 09:09:19 +08:00
Xinyi Zou
f90a1a1919 [fix](ut)(compile) Fix ut failure at functions_geo and compilation bug (#8843) 2022-04-05 21:30:40 +08:00
Gabriel
f3c6ddf651 [feature](function) Support geolocation functions on vectorized engine (#8790) 2022-04-03 10:50:54 +08:00
Lightman
6b0a642390 [feature][vectorized] Support explode json array func #8526 (#8539) 2022-04-03 10:06:47 +08:00
camby
4d516bece8 [feature-wip](array-type)Add element_at and subscript functions (#8597) 
Describe the overview of changes.
1. add function element_at;
2. support element_subscript([]) to get element of array, col_array[N] <==> element_at(col_array, N);
3. return error message instead of BE crash while array function execute failed;

element_at(array, index) desc:
>   Returns element of array at given **(1-based)** index. 
  If **index < 0**, accesses elements from the last to the first. 
  Returns NULL if the index exceeds the length of the array or the array is NULL.

Usage example:
1. create table with ARRAY type column and insert some data:
```
+------+------+--------+
| k1   | k2   | k3     |
+------+------+--------+
|    1 |    2 | [1, 2] |
|    2 |    3 | NULL   |
|    4 | NULL | []     |
|    3 | NULL | NULL   |
+------+------+--------+
```
2. enable vectorized:
```
set enable_vectorized_engine=true;
```
3. element_subscript([]) usage example:
```
> select k1,k3,k3[1] from array_test;
+------+--------+----------------------------+
| k1   | k3     | %element_extract%(`k3`, 1) |
+------+--------+----------------------------+
|    3 | NULL   |                       NULL |
|    1 | [1, 2] |                          1 |
|    2 | NULL   |                       NULL |
|    4 | []     |                       NULL |
+------+--------+----------------------------+
```
4. element_at function usage example:
```
> select k1,k3 from array_test where element_at(k3, -1) = 2;
+------+--------+
| k1   | k3     |
+------+--------+
|    1 | [1, 2] |
+------+--------+
```
 2022-04-02 12:03:56 +08:00
Gabriel
1ddfe20950 fix typo (#8714) 
fix typo
 2022-03-29 18:21:16 +08:00
wangbo
726eaa68ea [fix](vectorization) Vectorization decimal arithmetic inconsistent (#8626) 2022-03-28 10:12:39 +08:00
Gabriel
b89e4c7bba [feature-wip](java-udf) support java UDF with fixed-length input and output (#8516) 
This feature is propsoed in [DSIP-1](https://cwiki.apache.org/confluence/display/DORIS/DSIP-001%3A+Java+UDF). 
This PR support fixed-length input and output Java UDF. Phase I in DIP-1 is done after this PR.

To support Java UDF effeciently, I use no data copy in JNI call and all compute operations are off-heap in Java.
To achieve that, I use a UdfExecutor instead. 

For users, a UDF class must have a public evaluate method.
 2022-03-23 10:32:50 +08:00
camby
71ce3c4a6e [feature-wip](array-type) Add codes and UT for array_contains and array_position functions (#8401) (#8589) 
array_contains function Usage example:
1. create table with ARRAY column, and insert some data:
```
> select * from array_test;
+------+------+--------+
| k1   | k2   | k3     |
+------+------+--------+
|    1 |    2 | [1, 2] |
|    2 |    3 | NULL   |
|    4 | NULL | []     |
|    3 | NULL | NULL   |
+------+------+--------+
```
2. enable vectorized:
```
> set enable_vectorized_engine=true;
```
3. select with array_contains:
```
> select k1,array_contains(k3,1) from array_test;
+------+-------------------------+
| k1   | array_contains(`k3`, 1) |
+------+-------------------------+
|    3 |                    NULL |
|    1 |                       1 |
|    2 |                    NULL |
|    4 |                       0 |
+------+-------------------------+
```
4. also we can use array_contains in where condition
```
> select * from array_test where array_contains(k3,1);
+------+------+--------+
| k1   | k2   | k3     |
+------+------+--------+
|    1 |    2 | [1, 2] |
+------+------+--------+
```
5. array_position usage example
```
> select k1,k3,array_position(k3,2) from array_test;
+------+--------+-------------------------+
| k1   | k3     | array_position(`k3`, 2) |
+------+--------+-------------------------+
|    3 | NULL   |                    NULL |
|    1 | [1, 2] |                       2 |
|    2 | NULL   |                    NULL |
|    4 | []     |                       0 |
+------+--------+-------------------------+
```
 2022-03-22 15:42:40 +08:00
Pxl
be3d203289 [feature][vectorized] support table function explode_numbers() (#8509) 2022-03-22 11:38:00 +08:00
Pxl
fc3ad371c8 [fix](vec) fix regexp_replace get wrong result on clang (#8505) 2022-03-20 23:11:24 +08:00
Pxl
a8af8d2981 [fix](vectorized) fix core dump on get_json_string and add some ut (#8496) 2022-03-17 10:08:31 +08:00
Pxl
a824c3e489 [feature](vectorized) support lateral view (#8448) 2022-03-17 10:04:24 +08:00
zhangstar333
e0ef9b8f6c [refactor](vectorized) to_bitmap(-1) return NULL instead of return parse failed error_message (#8373) 2022-03-11 17:21:47 +08:00
Pxl
10c3712aa1 [fix](vectorized) fix arithmetic calculate get wrong result(#8226) 2022-03-09 13:03:57 +08:00
Pxl
cd8694e532 [feature][vectorized] support replace() (#8384) 2022-03-08 18:57:12 +08:00
zhangstar333
454b45bea3 [feature](vectorize)(function) support regexp&&sm4&&aes functions (#8307) 2022-03-08 13:14:02 +08:00
HappenLee
be4dd0c820 [fix][vectorized] Fix error cast to boolean (#8345) 2022-03-06 13:47:46 +08:00
Pxl
0ee53be883 [fix][improvement](runtime-filter) fix string type length limit error && add runtime filter decimal support (#8282) 2022-03-03 22:44:49 +08:00
zhangstar333
2b9b0fc1ec [Fix] Function percentile input null return null (#8238) 2022-03-01 14:42:48 +08:00
Pxl
668188b91f [improvement][vectorized] support es node predicate peel (#8174) 2022-02-26 17:02:54 +08:00
HappenLee
a6bc9cbe53 [Function] Refactor the function code of log (#8199) 
1. Support return null when input is invalid
2. Del the unless code in vec function

Co-authored-by: lihaopeng <lihaopeng@baidu.com>
 2022-02-24 11:06:58 +08:00
Pxl
90a8ca808a [Bug][Vectorized] fix bitmap_min(empty) not return null (#8190) 2022-02-24 11:06:27 +08:00
zhangstar333
31ab569c1d [Vectorized][Feature] support some bitmap functions (#8138) 2022-02-23 11:42:16 +08:00
Pxl
87e555c27d [Feature][Vectorized] support function json_array/json_object/json_quote (#8158) 2022-02-22 09:29:56 +08:00
Zhengguo Yang
c47368f80c [fix] (udf) fix check_fn and fn_call function name not same (#8132) 2022-02-22 09:18:07 +08:00
HappenLee
56adc7f56b [Bug][vec] Fix bug of nullable const value convert to argument cause coredump (#8139) 
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
 2022-02-20 20:05:23 +08:00
Zhengguo Yang
50864aca7d [refactor] fix warings when compile with clang (#8069) 2022-02-19 11:29:02 +08:00
HappenLee
bcde1f265a [Function][Vectorized] Support least/greast function (#8107) 
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
 2022-02-18 11:57:07 +08:00
HappenLee
f6e2a4fe16 [Vectorized][Function] Support year/month/week/hour/mintue/day/second floor/ceil function (#8068) 
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
 2022-02-17 14:18:02 +08:00
Pxl
f06c13a828 [feature](vec)(function) support function convert_tz() (#8060) 2022-02-17 10:51:32 +08:00
HappenLee
bef1b55c1f [feature][fix](vec)(function) Fix multi args function call the DATETIME type not effective in DATE type and add the alias function (#8050) 
1. Support some function alias of mod/fmod, adddate/add_data
2. Support some function of multi args: week, yearweek
3. Fix bug of multi args function call the DATETIME type not effective in DATE type
 2022-02-17 10:49:25 +08:00
zhangstar333
0003822da7 [feature](vec) add ColumnHLL to support hll type (#7828) 2022-02-17 10:44:42 +08:00
Pxl
143c4085ee [Feature][Vectorized] support aggregate function ndv()/approx_count_distinct() (#8044) 2022-02-16 14:30:13 +08:00
Pxl
b26e7e3c28 [feature](function)(vec) support locate function (#7988) 
* support function locate in vectorized engine

* add ut and fix some bug
 2022-02-12 16:00:37 +08:00
Pxl
64fb8dab39 [feature] (function)(vec) support pmod function (#7977) 2022-02-12 16:00:11 +08:00
Zhengguo Yang
5029ef46c9 [fix] fix ltrim result may incorrect in some case (#7963) 
fix ltrim result may incorrect in some case
according to https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
Built-in Function: int __builtin_cl/tz (unsigned int x)
If x is 0, the result is undefined.
So we handle the case of 0 separately

this function return different between gcc and clang when x is 0
 2022-02-09 13:06:37 +08:00
Zhengguo Yang
f8d086d87f [feature](rpc) (experimental)Support implement UDF through GRPC protocol. (#7519) 
Support implement UDF through GRPC protocol. This brings several benefits: 
1. The udf implementation language is not limited to c++, users can use any familiar language to implement udf
2. UDF is decoupled from Doris, udf will not cause doris coredump, udf computing resources are separated from doris, and doris services are not affected

But RPC's UDF has a fixed overhead, so its performance is much slower than C++ UDF, especially when the amount of data is large.

Create function like

```
CREATE FUNCTION rpc_add(INT, INT) RETURNS INT PROPERTIES (
  "SYMBOL"="add_int",
  "OBJECT_FILE"="127.0.0.1:9999",
  "TYPE"="RPC"
);
```
Function service need to implement `check_fn` and `fn_call` methods
Note:
THIS IS AN EXPERIMENTAL FEATURE, THE INTERFACE AND DATA STRUCTURE MAY BE CHANGED IN FUTURE !!!
 2022-02-08 09:25:09 +08:00
HappenLee
358bd79fb1 [improvement](vec)(Join) Mem reuse to speed up join operator (#7905) 
1. Reuse the mem of output block in vec join node
2. Add the function `replicate` in column
 2022-01-31 22:14:12 +08:00
Pxl
3ee000c13c [chore] support build with libc++ && add some build config (#7903) 
support LIBCPP/LDD/BUILD_META_TOOL for build.sh
 2022-01-30 16:47:22 +08:00
zhangstar333
fb6e22f4ca [Fix] fix memory leak in be unit test (#7857) 
1. fix be unit test memory leak
2. ignore mindump test with ASAN test
 2022-01-29 01:00:38 +08:00
zhangstar333
071be928f9 [fix](vectorized) fix bug multi distinct function get wrong type (#7900) 2022-01-28 22:31:41 +08:00
HappenLee
015371ac72 [fix](grouping-set) Fix the bug of grouping set core in both vec and non vec query engine (#7800) 2022-01-26 16:15:30 +08:00
zhangstar333
a6831535e9 [Vectorized][Bug] fix bug of coalesce function (#7827) 2022-01-25 20:44:16 +08:00