4960043f5e
[enhancement] Refactor to improve the usability of MemTracker (step2) ( #10823 )
2022-07-21 17:11:28 +08:00
dc2b709f6f
[Bug](compaction) fix uniq key compaction bug that does not count merged rows right( #10971 )
...
When a rowset includes multiple segments, segments rows will be merged in generic_iterator but merged_rows is not maintained. Compaction will failed in check_correctness.
Co-authored-by: yixiutt <yixiu@selectdb.com >
2022-07-20 12:07:45 +08:00
09d19e3f0f
[feature-wip](array-type) explode support more sub types ( #10673 )
...
1. explode support more sub types;
2. explode support nullable elements;
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-07-17 18:08:30 +08:00
5793cb11d0
[feature-wip] (array-type) function concat_ws support array ( #10749 )
...
Issue #10052
function concat_ws support array
2022-07-17 17:50:39 +08:00
00c9455f16
[fix](array-type) fix arrow column to doris array column ( #10855 )
...
* support merge array column, while convert from arrow column to doris array column
* fix typo
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-07-16 11:49:42 +08:00
dc6fbcce14
[feature-wip] (datev2) modify datev2 format in memory ( #10873 )
...
* [feature-wip] (datev2) modify datev2 format in memory
* update
2022-07-15 19:57:38 +08:00
1112dba525
[be ut]add some case for array type in block_test ( #10656 )
...
Co-authored-by: hucheng01 <hucheng01@baidu.com >
2022-07-09 12:00:42 +08:00
fe8acdb268
[feature-wip](array-type) add agg function collect_list and collect_set ( #10606 )
...
add codes for collect_list and collect_set and update regression output, before output format for ARRAY(string) already changed.
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-07-08 12:48:46 +08:00
8012d63ea0
[fix] substr('', 1, 5) return empty string instead of null ( #10622 )
2022-07-06 22:51:02 +08:00
1f57fcc4e9
remove duplicate codes from function_test_util.cpp ( #10607 )
...
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-07-05 20:43:56 +08:00
ec6620ae3e
[feature-wip](array-type) add function arrays_overlap ( #10233 )
2022-06-30 08:12:29 +08:00
ca94867b4e
[Feature-wip] add date v2 type ( #9916 )
2022-06-26 16:07:56 +08:00
5e47b03595
[feature-wip](array-type) Add array aggregation functions ( #10108 )
2022-06-17 11:07:49 +08:00
d58e00c49c
[fix](brpc) Embed serialized request into the attachment and transmit it through http brpc ( #9803 )
...
When the length of `Tuple/Block data` is greater than 2G, serialize the protoBuf request and embed the
`Tuple/Block data` into the controller attachment and transmit it through http brpc.
This is to avoid errors when the length of the protoBuf request exceeds 2G:
`Bad request, error_text=[E1003]Fail to compress request`.
In #7164 , `Tuple/Block data` was put into attachment and sent via default `baidu_std brpc`,
but when the attachment exceeds 2G, it will be truncated. There is no 2G limit for sending via `http brpc`.
Also, in #7921 , consider putting `Tuple/Block data` into attachment transport by default, as this theoretically
reduces one serialization and improves performance. However, the test found that the performance did not improve,
but the memory peak increased due to the addition of a memory copy.
2022-06-13 20:41:48 +08:00
6fab1cbf3c
[feature-wip](array-type) Add array functions size and cardinality ( #9921 )
...
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-06-09 15:03:03 +08:00
19bc14cf8d
[feature-wip](array-type) Add array type support for vectorized parquet-orc scanner ( #9856 )
...
Only support one level array now.
for example:
- nullable(array(nullable(tinyint))) is **support**.
- nullable(array(nullable(array(xx))) is **not support**.
2022-06-09 12:11:47 +08:00
94089b9192
[Refactor] Use file factory to replace create file reader/writer ( #9505 )
...
1. Simplify code logic and improve abstraction
2. Fix the mem leak of raw pointer
Co-authored-by: lihaopeng <lihaopeng@baidu.com >
2022-06-08 15:07:39 +08:00
c0ad1be1bd
[Enhancement][Chore] remove breakpad and unused variable ( #9937 )
2022-06-02 20:52:17 +08:00
632f7a3d3d
[Feature] add weekday function on vectorized engine ( #9901 )
2022-06-01 14:47:37 +08:00
0cba6b7d95
[Bug][Fix] One Rowset have same key output in unique table ( #9858 )
...
Co-authored-by: lihaopeng <lihaopeng@baidu.com >
2022-05-31 12:29:16 +08:00
f377c26bf7
[refactor][be] Optimize headers ( #9708 )
2022-05-30 16:12:10 +08:00
7b98dd438d
[feature](function) Add nvl function ( #9726 )
2022-05-30 09:43:00 +08:00
cbbda7857b
[feature-wip](parquet-orc) Support orc scanner in vectorized engine ( #9541 )
2022-05-26 21:39:12 +08:00
13c1d20426
[Bug] [Vectorized] add padding when load char type data ( #9734 )
2022-05-26 16:51:01 +08:00
8470543144
[Improvement] fix typo ( #9743 )
2022-05-25 19:29:01 +08:00
31e40191a8
[Refactor] add vpre_filter_expr for vectorized to improve performance ( #9508 )
2022-05-22 11:45:57 +08:00
8fa677b59c
[Refactor][Bug-Fix][Load Vec] Refactor code of basescanner and vjson/vparquet/vbroker scanner ( #9666 )
...
* [Refactor][Bug-Fix][Load Vec] Refactor code of basescanner and vjson/vparquet/vbroker scanner
1. fix bug of vjson scanner not support `range_from_file_path`
2. fix bug of vjson/vbrocker scanner core dump by src/dest slot nullable is different
3. fix bug of vparquest filter_block reference of column in not 1
4. refactor code to simple all the code
It only changed vectorized load, not original row based load.
Co-authored-by: lihaopeng <lihaopeng@baidu.com >
2022-05-20 11:43:03 +08:00
bee5c2f8aa
[feature-wip](parquet-vec) Support parquet scanner in vectorized engine ( #9433 )
2022-05-17 09:37:17 +08:00
953429e370
[fix](function) fix last_value get wrong result when have order by clause ( #9247 )
2022-05-15 23:56:01 +08:00
b817efd652
[feature] add vectorized vjson_scanner ( #9311 )
...
This pr is used to add the vectorized vjson_scanner, which can support vectorized json import in stream load flow.
2022-05-14 09:50:05 +08:00
650e3a6ba0
[feature-wip](array-type) array_contains support more nested data types ( #9170 )
...
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-05-13 12:42:40 +08:00
718a51a388
[refactor][style] Use clang-format to sort includes ( #9483 )
2022-05-10 21:25:35 +08:00
eec1dfde3a
[feature] (vec) instead of converting line to src tuple for stream load in vectorized. ( #9314 )
...
Co-authored-by: xiepengcheng01 <xiepengcheng01@xafj-palo-rpm64.xafj.baidu.com >
2022-05-09 11:24:07 +08:00
c9961c9bb9
[style] clang-format all c++ code ( #9305 )
...
- sh build-support/clang-format.sh to clang-format all c++ code
2022-04-29 16:14:22 +08:00
d330bc3806
[Vectorized](stream-load-vec) Support stream load in vectorized engine ( #8709 ) ( #9280 )
...
Implement vectorized stream load.
Added fe configuration option `enable_vectorized_load` to enable vectorized stream load.
Co-authored-by: tengjp@outlook.com
Co-authored-by: mrhhsg@gmail.com
Co-authored-by: minghong.zhou@163.com
Co-authored-by: HappenLee <happenlee@hotmail.com >
Co-authored-by: zhoubintao <35688959+zbtzbtzbt@users.noreply.github.com >
2022-04-29 09:50:51 +08:00
2d83167e50
[Feature] [Lateral-View] support outer combinator of table function ( #9147 )
2022-04-24 12:09:40 +08:00
a2edc6fd8b
[feature-wip](array-type) replicate impl for ColumnArray to support join with array column ( #9070 )
...
SQL with JOIN and columns ARRAY, will call function ColumnArray::replicate. At this pr,
we implement replicate for ARRAY type, to support SQL like this:
`SELECT count(lo_array),count(d_array),SUM(lo_extendedprice*lo_discount) AS REVENUE FROM lineorder, date WHERE lo_orderdate = d_datekey AND d_year = 1993 AND lo_discount BETWEEN 1 AND 3 AND lo_quantity < 25;`
2022-04-20 14:50:34 +08:00
1b4cd76847
[feature](vectorized)(function) Support min_by/max_by function. ( #8623 )
...
Support min_by/max_by on vectorized engine.
2022-04-20 14:46:19 +08:00
52d18aa83c
permute impl for column array; and codes format ( #8949 )
...
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com >
2022-04-13 09:47:54 +08:00
5a44eeaf62
[refactor] Unify all unit tests into one binary file ( #8958 )
...
1. solved the previous delayed unit test file size is too large (1.7G+) and the unit test link time is too long problem problems
2. Unify all unit tests into one file to significantly reduce unit test execution time to less than 3 mins
3. temporarily disable stream_load_test.cpp, metrics_action_test.cpp, load_channel_mgr_test.cpp because it will re-implement part of the code and affect other tests
2022-04-12 15:30:40 +08:00
7f7172807f
[feature](function)(vectorized) Support all geolocation functions on vectorized engine ( #8846 )
2022-04-11 09:36:53 +08:00
c5718928df
[feature-wip](array-type) support explode and explode_outer table function ( #8766 )
...
explode(ArrayColumn) desc:
> Create a row for each element in the array column.
explode_outer(ArrayColumn) desc:
> Create a row for each element in the array column. Unlike explode, if the array is null or empty, it returns null.
Usage example:
1. create a table with array column, and insert some data;
2. open enable_lateral_view and enable_vectorized_engine;
```
set enable_lateral_view = true;
set enable_vectorized_engine=true;
```
3. use explode_outer
```
> select * from array_test;
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 3 | NULL | NULL |
| 1 | 2 | [1, 2] |
| 2 | 3 | NULL |
| 4 | NULL | [] |
+------+------+--------+
> select k1,explode_column from array_test LATERAL VIEW explode_outer(k3) TempExplodeView as explode_column;
+------+----------------+
| k1 | explode_column |
+------+----------------+
| 1 | 1 |
| 1 | 2 |
| 2 | NULL |
| 4 | NULL |
| 3 | NULL |
+------+----------------+
```
4. explode usage example. explode return empty rows while the ARRAY is null or empty
```
> select k1,explode_column from array_test LATERAL VIEW explode(k3) TempExplodeView as explode_column;
+------+----------------+
| k1 | explode_column |
+------+----------------+
| 1 | 1 |
| 1 | 2 |
+------+----------------+
```
2022-04-08 12:11:04 +08:00
f3c6ddf651
[feature](function) Support geolocation functions on vectorized engine ( #8790 )
2022-04-03 10:50:54 +08:00
a75e4a1469
Window funnel ( #8485 )
...
Add new feature window funnel
2022-04-02 22:08:50 +08:00
4d516bece8
[feature-wip](array-type)Add element_at and subscript functions ( #8597 )
...
Describe the overview of changes.
1. add function element_at;
2. support element_subscript([]) to get element of array, col_array[N] <==> element_at(col_array, N);
3. return error message instead of BE crash while array function execute failed;
element_at(array, index) desc:
> Returns element of array at given **(1-based)** index.
If **index < 0**, accesses elements from the last to the first.
Returns NULL if the index exceeds the length of the array or the array is NULL.
Usage example:
1. create table with ARRAY type column and insert some data:
```
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 1 | 2 | [1, 2] |
| 2 | 3 | NULL |
| 4 | NULL | [] |
| 3 | NULL | NULL |
+------+------+--------+
```
2. enable vectorized:
```
set enable_vectorized_engine=true;
```
3. element_subscript([]) usage example:
```
> select k1,k3,k3[1] from array_test;
+------+--------+----------------------------+
| k1 | k3 | %element_extract%(`k3`, 1) |
+------+--------+----------------------------+
| 3 | NULL | NULL |
| 1 | [1, 2] | 1 |
| 2 | NULL | NULL |
| 4 | [] | NULL |
+------+--------+----------------------------+
```
4. element_at function usage example:
```
> select k1,k3 from array_test where element_at(k3, -1) = 2;
+------+--------+
| k1 | k3 |
+------+--------+
| 1 | [1, 2] |
+------+--------+
```
2022-04-02 12:03:56 +08:00
71ce3c4a6e
[feature-wip](array-type) Add codes and UT for array_contains and array_position functions ( #8401 ) ( #8589 )
...
array_contains function Usage example:
1. create table with ARRAY column, and insert some data:
```
> select * from array_test;
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 1 | 2 | [1, 2] |
| 2 | 3 | NULL |
| 4 | NULL | [] |
| 3 | NULL | NULL |
+------+------+--------+
```
2. enable vectorized:
```
> set enable_vectorized_engine=true;
```
3. select with array_contains:
```
> select k1,array_contains(k3,1) from array_test;
+------+-------------------------+
| k1 | array_contains(`k3`, 1) |
+------+-------------------------+
| 3 | NULL |
| 1 | 1 |
| 2 | NULL |
| 4 | 0 |
+------+-------------------------+
```
4. also we can use array_contains in where condition
```
> select * from array_test where array_contains(k3,1);
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 1 | 2 | [1, 2] |
+------+------+--------+
```
5. array_position usage example
```
> select k1,k3,array_position(k3,2) from array_test;
+------+--------+-------------------------+
| k1 | k3 | array_position(`k3`, 2) |
+------+--------+-------------------------+
| 3 | NULL | NULL |
| 1 | [1, 2] | 2 |
| 2 | NULL | NULL |
| 4 | [] | 0 |
+------+--------+-------------------------+
```
2022-03-22 15:42:40 +08:00
a498463ab5
[feature-wip](array-type)support select ARRAY data type on vectorized engine ( #8217 ) ( #8584 )
...
Usage Example:
1. create table for test;
```
`CREATE TABLE `array_test` (
`k1` tinyint(4) NOT NULL COMMENT "",
`k2` smallint(6) NULL COMMENT "",
`k3` ARRAY<int(11)> NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`k1`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`k1`) BUCKETS 5
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
);`
```
2. insert some data
```
`insert into array_test values(1, 2, [1, 2]);`
`insert into array_test values(2, 3, null);`
`insert into array_test values(3, null, null);`
`insert into array_test values(4, null, []);`
```
3. open vectorized
`set enable_vectorized_engine=true;`
4. query array data
`select * from array_test;`
+------+------+--------+
| k1 | k2 | k3 |
+------+------+--------+
| 4 | NULL | [] |
| 2 | 3 | NULL |
| 1 | 2 | [1, 2] |
| 3 | NULL | NULL |
+------+------+--------+
4 rows in set (0.061 sec)
Code Changes include：
1. add column_array, data_type_array codes;
2. codes about data_type creation by Field, TabletColumn, TypeDescriptor, PColumnMeta move to DataTypeFactory;
3. support create data_type for ARRAY date type;
4. RowBlockV2::convert_to_vec_block support ARRAY date type;
5. VMysqlResultWriter::append_block support ARRAY date type;
6. vectorized::Block serialize and deserialize support ARRAY date type;
2022-03-22 15:21:44 +08:00
eeae516e37
[Feature](Memory) Hook TCMalloc new/delete automatically counts to MemTracker ( #8476 )
...
Early Design Documentation: https://shimo.im/docs/DT6JXDRkdTvdyV3G
Implement a new way of memory statistics based on TCMalloc New/Delete Hook,
MemTracker and TLS, and it is expected that all memory new/delete/malloc/free
of the BE process can be counted.
2022-03-20 23:06:54 +08:00
a8af8d2981
[fix](vectorized) fix core dump on get_json_string and add some ut ( #8496 )
2022-03-17 10:08:31 +08:00
febfe2f09d
[improvement](ut) add unit tests for min/max function, and cleaned up some unused code ( #8458 )
2022-03-15 11:43:18 +08:00
