f329d33666
[chore](fix) Fix some spell errors in be's comments. #13452
2022-10-20 08:56:01 +08:00
c449028a5f
[fix](year) fix year() results are not as expected ( #13426 )
...
fix `year()` results are not as expected
2022-10-19 11:28:00 +08:00
755a946516
[feature](jsonb) jsonb functions ( #13366 )
...
Issue Number: Step3 of DSIP-016: Support JSON type
2022-10-19 08:44:08 +08:00
f0dbbe5b46
[Bug](funciton) fix repeat coredump when step is to long ( #13408 )
2022-10-18 09:55:06 +08:00
1d5ba9cbcc
[Improvement](like) Change like function to batch call ( #13314 )
2022-10-16 16:18:22 +08:00
1bd14f1d82
[feature-wip](jsonb) jsonb parse function and load ( #13129 )
...
add function to parse json string to jsonb format and use it to support stream load.
2022-10-12 13:56:37 +08:00
16999ef02d
[Vectorized][Function] support date_trunc and countequal function ( #13039 )
2022-10-12 10:01:09 +08:00
1bb42a7bc0
[function](hash) add support of murmur_hash3_64 ( #12923 )
2022-09-26 14:23:37 +08:00
0ead048b93
[Enhancement](column) remove ColumnString terminating zero and add a data_version for pblock ( #12456 )
...
1. remove ColumnString terminating zero
2. add a data_version for pblock
3. change EncryptionMode to enum class
2022-09-14 21:25:22 +08:00
c8e9a32bb2
[Function](cbrt)Add cbrt function for doris ( #12523 )
...
Add cbrt function for doris
2022-09-12 19:58:45 +08:00
09b45f2b71
[Function](ELT)Add elt function ( #12321 )
2022-09-07 15:21:08 +08:00
328a225050
[feature-wip] (datetimev2) support window funnel and modify valid dat… ( #11277 )
...
* [feature-wip] (datetimev2) support window funnel and modify valid date range
2022-07-28 14:06:26 +08:00
72d2feae99
[feature-wip] Support all date functions for datev2/datetimev2 ( #11265 )
...
* [feature-wip] (datetimev2) support convert_tz function
* [feature-wip] Support all date functions for datev2/datetimev2
2022-07-28 08:18:59 +08:00
823088a9eb
[FOLLOW-UP] (datetimev2) complete date function ut and built-in function declaration ( #11154 )
2022-07-26 17:48:57 +08:00
babab5d535
[feature-wip] support datetimev2 ( #11085 )
2022-07-23 16:07:59 +08:00
09d19e3f0f
[feature-wip](array-type) explode support more sub types ( #10673 )
...
1. explode support more sub types;
2. explode support nullable elements;
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-07-17 18:08:30 +08:00
5793cb11d0
[feature-wip] (array-type) function concat_ws support array ( #10749 )
...
Issue #10052
function concat_ws support array
2022-07-17 17:50:39 +08:00
8012d63ea0
[fix] substr('', 1, 5) return empty string instead of null ( #10622 )
2022-07-06 22:51:02 +08:00
1f57fcc4e9
remove duplicate codes from function_test_util.cpp ( #10607 )
...
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-07-05 20:43:56 +08:00
ec6620ae3e
[feature-wip](array-type) add function arrays_overlap ( #10233 )
2022-06-30 08:12:29 +08:00
ca94867b4e
[Feature-wip] add date v2 type ( #9916 )
2022-06-26 16:07:56 +08:00
5e47b03595
[feature-wip](array-type) Add array aggregation functions ( #10108 )
2022-06-17 11:07:49 +08:00
6fab1cbf3c
[feature-wip](array-type) Add array functions size and cardinality ( #9921 )
...
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-06-09 15:03:03 +08:00
632f7a3d3d
[Feature] add weekday function on vectorized engine ( #9901 )
2022-06-01 14:47:37 +08:00
f377c26bf7
[refactor][be] Optimize headers ( #9708 )
2022-05-30 16:12:10 +08:00
7b98dd438d
[feature](function) Add nvl function ( #9726 )
2022-05-30 09:43:00 +08:00
8470543144
[Improvement] fix typo ( #9743 )
2022-05-25 19:29:01 +08:00
953429e370
[fix](function) fix last_value get wrong result when have order by clause ( #9247 )
2022-05-15 23:56:01 +08:00
650e3a6ba0
[feature-wip](array-type) array_contains support more nested data types ( #9170 )
...
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-05-13 12:42:40 +08:00
2d83167e50
[Feature] [Lateral-View] support outer combinator of table function ( #9147 )
2022-04-24 12:09:40 +08:00
5a44eeaf62
[refactor] Unify all unit tests into one binary file ( #8958 )
...
1. solved the previous delayed unit test file size is too large (1.7G+) and the unit test link time is too long problem problems
2. Unify all unit tests into one file to significantly reduce unit test execution time to less than 3 mins
3. temporarily disable stream_load_test.cpp, metrics_action_test.cpp, load_channel_mgr_test.cpp because it will re-implement part of the code and affect other tests
2022-04-12 15:30:40 +08:00
7f7172807f
[feature](function)(vectorized) Support all geolocation functions on vectorized engine ( #8846 )
2022-04-11 09:36:53 +08:00
c5718928df
[feature-wip](array-type) support explode and explode_outer table function ( #8766 )
...
explode(ArrayColumn) desc:
> Create a row for each element in the array column.
explode_outer(ArrayColumn) desc:
> Create a row for each element in the array column. Unlike explode, if the array is null or empty, it returns null.
Usage example:
1. create a table with array column, and insert some data;
2. open enable_lateral_view and enable_vectorized_engine;
```
set enable_lateral_view = true;
set enable_vectorized_engine=true;
```
3. use explode_outer
```
> select * from array_test;
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 3 | NULL | NULL |
| 1 | 2 | [1, 2] |
| 2 | 3 | NULL |
| 4 | NULL | [] |
+------+------+--------+
> select k1,explode_column from array_test LATERAL VIEW explode_outer(k3) TempExplodeView as explode_column;
+------+----------------+
| k1 | explode_column |
+------+----------------+
| 1 | 1 |
| 1 | 2 |
| 2 | NULL |
| 4 | NULL |
| 3 | NULL |
+------+----------------+
```
4. explode usage example. explode return empty rows while the ARRAY is null or empty
```
> select k1,explode_column from array_test LATERAL VIEW explode(k3) TempExplodeView as explode_column;
+------+----------------+
| k1 | explode_column |
+------+----------------+
| 1 | 1 |
| 1 | 2 |
+------+----------------+
```
2022-04-08 12:11:04 +08:00
f3c6ddf651
[feature](function) Support geolocation functions on vectorized engine ( #8790 )
2022-04-03 10:50:54 +08:00
4d516bece8
[feature-wip](array-type)Add element_at and subscript functions ( #8597 )
...
Describe the overview of changes.
1. add function element_at;
2. support element_subscript([]) to get element of array, col_array[N] <==> element_at(col_array, N);
3. return error message instead of BE crash while array function execute failed;
element_at(array, index) desc:
> Returns element of array at given **(1-based)** index.
If **index < 0**, accesses elements from the last to the first.
Returns NULL if the index exceeds the length of the array or the array is NULL.
Usage example:
1. create table with ARRAY type column and insert some data:
```
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 1 | 2 | [1, 2] |
| 2 | 3 | NULL |
| 4 | NULL | [] |
| 3 | NULL | NULL |
+------+------+--------+
```
2. enable vectorized:
```
set enable_vectorized_engine=true;
```
3. element_subscript([]) usage example:
```
> select k1,k3,k3[1] from array_test;
+------+--------+----------------------------+
| k1 | k3 | %element_extract%(`k3`, 1) |
+------+--------+----------------------------+
| 3 | NULL | NULL |
| 1 | [1, 2] | 1 |
| 2 | NULL | NULL |
| 4 | [] | NULL |
+------+--------+----------------------------+
```
4. element_at function usage example:
```
> select k1,k3 from array_test where element_at(k3, -1) = 2;
+------+--------+
| k1 | k3 |
+------+--------+
| 1 | [1, 2] |
+------+--------+
```
2022-04-02 12:03:56 +08:00
71ce3c4a6e
[feature-wip](array-type) Add codes and UT for array_contains and array_position functions ( #8401 ) ( #8589 )
...
array_contains function Usage example:
1. create table with ARRAY column, and insert some data:
```
> select * from array_test;
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 1 | 2 | [1, 2] |
| 2 | 3 | NULL |
| 4 | NULL | [] |
| 3 | NULL | NULL |
+------+------+--------+
```
2. enable vectorized:
```
> set enable_vectorized_engine=true;
```
3. select with array_contains:
```
> select k1,array_contains(k3,1) from array_test;
+------+-------------------------+
| k1 | array_contains(`k3`, 1) |
+------+-------------------------+
| 3 | NULL |
| 1 | 1 |
| 2 | NULL |
| 4 | 0 |
+------+-------------------------+
```
4. also we can use array_contains in where condition
```
> select * from array_test where array_contains(k3,1);
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 1 | 2 | [1, 2] |
+------+------+--------+
```
5. array_position usage example
```
> select k1,k3,array_position(k3,2) from array_test;
+------+--------+-------------------------+
| k1 | k3 | array_position(`k3`, 2) |
+------+--------+-------------------------+
| 3 | NULL | NULL |
| 1 | [1, 2] | 2 |
| 2 | NULL | NULL |
| 4 | [] | 0 |
+------+--------+-------------------------+
```
2022-03-22 15:42:40 +08:00
eeae516e37
[Feature](Memory) Hook TCMalloc new/delete automatically counts to MemTracker ( #8476 )
...
Early Design Documentation: https://shimo.im/docs/DT6JXDRkdTvdyV3G
Implement a new way of memory statistics based on TCMalloc New/Delete Hook,
MemTracker and TLS, and it is expected that all memory new/delete/malloc/free
of the BE process can be counted.
2022-03-20 23:06:54 +08:00
a8af8d2981
[fix](vectorized) fix core dump on get_json_string and add some ut ( #8496 )
2022-03-17 10:08:31 +08:00
e17aef9467
[refactor] refactor the implement of MemTracker, and related usage ( #8322 )
...
Modify the implementation of MemTracker:
1. Simplify a lot of useless logic;
2. Added MemTrackerTaskPool, as the ancestor of all query and import trackers, This is used to track the local memory usage of all tasks executing;
3. Add cosume/release cache, trigger a cosume/release when the memory accumulation exceeds the parameter mem_tracker_consume_min_size_bytes;
4. Add a new memory leak detection mode (Experimental feature), throw an exception when the remaining statistical value is greater than the specified range when the MemTracker is destructed, and print the accurate statistical value in HTTP, the parameter memory_leak_detection
5. Added Virtual MemTracker, cosume/release will not sync to parent. It will be used when introducing TCMalloc Hook to record memory later, to record the specified memory independently;
6. Modify the GC logic, register the buffer cached in DiskIoMgr as a GC function, and add other GC functions later;
7. Change the global root node from Root MemTracker to Process MemTracker, and remove Process MemTracker in exec_env;
8. Modify the macro that detects whether the memory has reached the upper limit, modify the parameters and default behavior of creating MemTracker, modify the error message format in mem_limit_exceeded, extend and apply transfer_to, remove Metric in MemTracker, etc.;
Modify where MemTracker is used:
1. MemPool adds a constructor to create a temporary tracker to avoid a lot of redundant code;
2. Added trackers for global objects such as ChunkAllocator and StorageEngine;
3. Added more fine-grained trackers such as ExprContext;
4. RuntimeState removes FragmentMemTracker, that is, PlanFragmentExecutor mem_tracker, which was previously used for independent statistical scan process memory, and replaces it with _scanner_mem_tracker in OlapScanNode;
5. MemTracker is no longer recorded in ReservationTracker, and ReservationTracker will be removed later;
2022-03-11 22:04:23 +08:00
e0ef9b8f6c
[refactor](vectorized) to_bitmap(-1) return NULL instead of return parse failed error_message ( #8373 )
2022-03-11 17:21:47 +08:00
cd8694e532
[feature][vectorized] support replace() ( #8384 )
2022-03-08 18:57:12 +08:00
454b45bea3
[feature](vectorize)(function) support regexp&&sm4&&aes functions ( #8307 )
2022-03-08 13:14:02 +08:00
2b9b0fc1ec
[Fix] Function percentile input null return null ( #8238 )
2022-03-01 14:42:48 +08:00
7d0e36a054
[fix](be-ut) fix bitmap_ut result wrong && fix schema_change compile error ( #8261 )
2022-03-01 11:11:02 +08:00
a6bc9cbe53
[Function] Refactor the function code of log ( #8199 )
...
1. Support return null when input is invalid
2. Del the unless code in vec function
Co-authored-by: lihaopeng <lihaopeng@baidu.com >
2022-02-24 11:06:58 +08:00
31ab569c1d
[Vectorized][Feature] support some bitmap functions ( #8138 )
2022-02-23 11:42:16 +08:00
bcde1f265a
[Function][Vectorized] Support least/greast function ( #8107 )
...
Co-authored-by: lihaopeng <lihaopeng@baidu.com >
2022-02-18 11:57:07 +08:00
f06c13a828
[feature](vec)(function) support function convert_tz() ( #8060 )
2022-02-17 10:51:32 +08:00
bef1b55c1f
[feature][fix](vec)(function) Fix multi args function call the DATETIME type not effective in DATE type and add the alias function ( #8050 )
...
1. Support some function alias of mod/fmod, adddate/add_data
2. Support some function of multi args: week, yearweek
3. Fix bug of multi args function call the DATETIME type not effective in DATE type
2022-02-17 10:49:25 +08:00
b26e7e3c28
[feature](function)(vec) support locate function ( #7988 )
...
* support function locate in vectorized engine
* add ut and fix some bug
2022-02-12 16:00:37 +08:00
