Commit Graph

31 Commits

Author SHA1 Message Date
f7988fad03 [improvement](string) set bigger limit for ColumnString chars length (#15426) 2022-12-28 15:41:01 +08:00
301640d3c0 [fix](string) fix offsets over flow for extreme large String column (#15360)
* [fix](string) fix offsets over flow for extreme large String column

* fix
2022-12-26 21:23:58 +08:00
62f765b7f5 [improvement](scan) speed up inserting strings into ColumnString (#13397) 2022-11-02 22:19:02 +08:00
cd3450bd9d [Improvement](join) optimize join probing phase (#13357) 2022-10-18 12:37:17 +08:00
830183984a [fix](hash)update_hashes_with_value method should handle if input value is null (#13332)
* [fix](hash)update_hashes_with_value method should handle if input value is null

* remove unnessasery xxHash64NullWithSeed
2022-10-13 14:36:01 +08:00
34b14a71c8 [Improvement](string) Optimize scanning for string #12911
~0.2X performance boost for queries containing string predicates
2022-09-29 15:11:16 +08:00
35076431ab [fix](column)fix get_shrinked_column misspell (#12961)
Fix misspell
2022-09-26 17:32:03 +08:00
f879a51ce9 [Improvement](dict) optimize dictionary column (#12852) 2022-09-25 18:29:10 +08:00
32551a7263 [bugfix](predicate column) data maybe wrong if not a single page (#12796)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2022-09-22 09:55:31 +08:00
3cfaae0031 [Improvement](sort) Use heap sort to optimize sort node (#12700) 2022-09-21 10:01:52 +08:00
35b97a5af0 [Opt](hash) Speed up insert from dict data map and not datetime (#12670)
Speed up dict data read and not datetime. same target #12636
2022-09-17 17:02:43 +08:00
Pxl
d44ec74988 [Enhancement](column) optimize for ColumnString::insert_many_dict_data (#12636)
optimize for ColumnString::insert_many_dict_data
2022-09-16 10:23:04 +08:00
e413a2b8e9 [Opt](vectorized) Use new way to do hash shffle to speed up query (#12586) 2022-09-15 11:08:04 +08:00
Pxl
0ead048b93 [Enhancement](column) remove ColumnString terminating zero and add a data_version for pblock (#12456)
1. remove ColumnString terminating zero
    2. add a data_version for pblock
    3. change EncryptionMode to enum class
2022-09-14 21:25:22 +08:00
56b2fc43d4 [enhancement](array-type) shrink column suffix zero for type ARRAY<CHAR> (#12443)
In compute level, CHAR type will shrink suffix zeros.
To keep the logic the same as CHAR type, we also shrink for ARRAY or ARRAY<ARRAY> types.

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-09-13 23:24:48 +08:00
d913ca5731 [Opt](vectorized) Speed up bucket shuffle join hash compute (#12407)
* [Opt](vectorized) Speed up bucket shuffle join hash compute
2022-09-13 20:19:22 +08:00
66491ec137 [Improvement](sort) improve partial sort algorithm (#12349)
* [Improvement](sort) improve partial sort algorithm
2022-09-09 15:44:18 +08:00
54d1630c42 [Opt](vectorized) speed up hash function compute in hash partition (#12334)
After do the opt of hash function, the compute of siphash in HASH_PARTITION in vdata_stream_sender

Before: 1s800ms
After: 800ms
2022-09-07 10:11:40 +08:00
90fb3b7783 [Improvement](load) accelerate tablet sink (#12174) 2022-09-01 10:08:09 +08:00
b7c9007776 [improvement][agg]Process aggregated results in the vectorized way (#11084) 2022-07-22 22:04:43 +08:00
e293fbd277 [improvement]pre-serialize aggregation keys (#10700) 2022-07-09 06:21:56 +08:00
Pxl
a9d23ce337 [refactor] remove collator (#10518) 2022-07-01 10:35:32 +08:00
18ad8ebfbb [improvement]Add reading by rowids to speed up lazy materialization (#10506) 2022-06-30 21:03:41 +08:00
Pxl
13c1d20426 [Bug] [Vectorized] add padding when load char type data (#9734) 2022-05-26 16:51:01 +08:00
Pxl
7fc22c2456 [fix][vectorized] fix core on get_predicate_column_ptr && fix double copy on _read_columns_by_rowids (#8581) 2022-03-24 09:12:42 +08:00
2ec0b81030 [improvement](storage) Low cardinality string optimization in storage layer (#8318)
Low cardinality string optimization in storage layer
2022-03-20 23:04:25 +08:00
802fcbbb05 (#8162)refactor binary dict
Co-authored-by: zuochunwei <zuochunwei@meituan.com>
2022-02-22 11:23:54 +08:00
b9f0b5565c [refactor](storage) refactor some interfaces of storage layer column (#8064)
1 format binary plain
2 remove batch_set_null_bitmap
3 fix segiter return value
4 set insert_many_binary_data args
2022-02-18 10:54:51 +08:00
358bd79fb1 [improvement](vec)(Join) Mem reuse to speed up join operator (#7905)
1. Reuse the mem of output block in vec join node
2. Add the function `replicate` in column
2022-01-31 22:14:12 +08:00
a6831535e9 [Vectorized][Bug] fix bug of coalesce function (#7827) 2022-01-25 20:44:16 +08:00
e1d7233e9c [feature](vectorization) Support Vectorized Exec Engine In Doris (#7785)
# Proposed changes

Issue Number: close #6238

    Co-authored-by: HappenLee <happenlee@hotmail.com>
    Co-authored-by: stdpain <34912776+stdpain@users.noreply.github.com>
    Co-authored-by: Zhengguo Yang <yangzhgg@gmail.com>
    Co-authored-by: wangbo <506340561@qq.com>
    Co-authored-by: emmymiao87 <522274284@qq.com>
    Co-authored-by: Pxl <952130278@qq.com>
    Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
    Co-authored-by: thinker <zchw100@qq.com>
    Co-authored-by: Zeno Yang <1521564989@qq.com>
    Co-authored-by: Wang Shuo <wangshuo128@gmail.com>
    Co-authored-by: zhoubintao <35688959+zbtzbtzbt@users.noreply.github.com>
    Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
    Co-authored-by: xinghuayu007 <1450306854@qq.com>
    Co-authored-by: weizuo93 <weizuo@apache.org>
    Co-authored-by: yiguolei <guoleiyi@tencent.com>
    Co-authored-by: anneji-dev <85534151+anneji-dev@users.noreply.github.com>
    Co-authored-by: awakeljw <993007281@qq.com>
    Co-authored-by: taberylyang <95272637+taberylyang@users.noreply.github.com>
    Co-authored-by: Cui Kaifeng <48012748+azurenake@users.noreply.github.com>


## Problem Summary:

### 1. Some code from clickhouse

**ClickHouse is an excellent implementation of the vectorized execution engine database,
so here we have referenced and learned a lot from its excellent implementation in terms of
data structure and function implementation.
We are based on ClickHouse v19.16.2.2 and would like to thank the ClickHouse community and developers.**

The following comment has been added to the code from Clickhouse, eg:
// This file is copied from
// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/AggregationCommon.h
// and modified by Doris

### 2. Support exec node and query:
* vaggregation_node
* vanalytic_eval_node
* vassert_num_rows_node
* vblocking_join_node
* vcross_join_node
* vempty_set_node
* ves_http_scan_node
* vexcept_node
* vexchange_node
* vintersect_node
* vmysql_scan_node
* vodbc_scan_node
* volap_scan_node
* vrepeat_node
* vschema_scan_node
* vselect_node
* vset_operation_node
* vsort_node
* vunion_node
* vhash_join_node

You can run exec engine of SSB/TPCH and 70% TPCDS stand query test set.

### 3. Data Model

Vec Exec Engine Support **Dup/Agg/Unq** table, Support Block Reader Vectorized.
Segment Vec is working in process.

### 4. How to use

1. Set the environment variable `set enable_vectorized_engine = true; `(required)
2. Set the environment variable `set batch_size = 4096; ` (recommended)

### 5. Some diff from origin exec engine

https://github.com/doris-vectorized/doris-vectorized/issues/294

## Checklist(Required)

1. Does it affect the original behavior: (No)
2. Has unit tests been added: (Yes)
3. Has document been added or modified: (No)
4. Does it need to update dependencies: (No)
5. Are there any changes that cannot be rolled back: (Yes)
2022-01-18 10:07:15 +08:00