Commit Graph

2151 Commits

Author SHA1 Message Date
1220cc147d [feature](vectorized) Support outfile on vectorized engine (#10013)
This PR supports output csv format file on vectorized engine.

** Parquet is still not supported. **
2022-06-10 09:15:53 +08:00
3363b3aa19 [fix](load) fix streamload failure due to false unhealthy replica in concurrent stream load (#10007)
in concurrent stream load, fe will run publish version task concurrently,
which cause publish task disorder in be.
For example:
fe publish task with version 1 2 3 4
be may handle task with sequence 1 2 4 3
In case above, when report tablet info, be found that version 4
published but version 3 not visible, it'll report version miss to fe,
and fe will set replica lastFailedVersion, and finally makes transaction
commits fail while no quorum health replicas。

Add a time condition if a version miss for 60 seconds then report version miss.
2022-06-10 09:15:14 +08:00
d247d06180 [Improvement] refine codes in TabletReader (#10042)
fast merge: just remove some code have no affect to other components
2022-06-10 09:12:33 +08:00
9c1ba771da [fix](be) fix asan be set_storage_medium core (#9986) (#9987)
https://github.com/apache/incubator-doris/issues/9986
2022-06-09 23:12:58 +08:00
dc874709d7 [feature-wip](array-type) support array<decimal128> in mysql_result_writer (#9998) 2022-06-09 15:15:26 +08:00
6fab1cbf3c [feature-wip](array-type) Add array functions size and cardinality (#9921)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-06-09 15:03:03 +08:00
19bc14cf8d [feature-wip](array-type) Add array type support for vectorized parquet-orc scanner (#9856)
Only support one level array now.
for example:
- nullable(array(nullable(tinyint))) is **support**.
- nullable(array(nullable(array(xx))) is **not support**.
2022-06-09 12:11:47 +08:00
bf8b4fb2d3 [Bugfix] be crash when executing sql contains bitmap_intersect function (#9910)
* fix bitmap serialize bug

* add regression test for bitmap seralize bugfix

* add missing regression test out file

* fix reggresion test failed issue
2022-06-09 08:45:46 +08:00
9c52b4a508 [enhance] improve dict in-predicate evaluate (#10009) 2022-06-09 00:25:30 +08:00
d9bbf67b9e [DefaultConfigChange]enable query vectorization and storage vectorization and storage low cardinality optimization by default (#9848)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2022-06-08 15:29:43 +08:00
94089b9192 [Refactor] Use file factory to replace create file reader/writer (#9505)
1. Simplify code logic and improve abstraction
2. Fix the mem leak of raw pointer

Co-authored-by: lihaopeng <lihaopeng@baidu.com>
2022-06-08 15:07:39 +08:00
fc9afda97a [enhancement][diagnostics] Add a diagnostic: detect unused includes (#9117) 2022-06-08 11:52:48 +08:00
35c3e4e33c [Bug] runtime filter is not used as expected (#10001)
* [Bug] runtime filter is not used as expected

* update
2022-06-08 11:10:39 +08:00
Pxl
f2aa5f32b8 [Feature] [Vectorized] Some pre-refactorings or interface additions for schema change (#9811)
Some pre-refactorings or interface additions for schema change
2022-06-07 15:04:57 +08:00
d588e99b8b [fix][mem tracker] Fix logout load task mem tracker dcheck fail (#9943)
* fix tracker 0602

* fix format
2022-06-07 11:31:49 +08:00
49d4798276 [fix](function) fix bug in time_round function (#9712) 2022-06-06 08:58:22 +08:00
79620f6fa2 [Improvement] change the condition of vectorized compaction (#9950) 2022-06-04 12:29:23 +08:00
3743f19369 [feature] support convert alpha rowset (#9890)
Add alpha rowset to beta rowset convert to convert rowset automatically. We will remove alpha rowset's code after 1.1.
2022-06-04 12:29:03 +08:00
f49284036e [Enhancement] Refactor functions in int_exp by templates (#9939) 2022-06-04 11:53:31 +08:00
c996334ad1 [improvement] Optimize send fragment logic to reduce send fragment timeout error (#9720)
This CL mainly changes:
1. Reducing the rpc timeout problem caused by rpc waiting for the worker thread of brpc.
    1. Merge multiple fragment instances on the same BE to send requests to reduce the number of send fragment rpcs
    2. If fragments size >= 3, use 2 phase RPC: one is to send all fragments, two is to start these fragments. So that there
         will be at most 2 RPC for each query on one BE.

3. Set the timeout of send fragment rpc to the query timeout to ensure the consistency of users' expectation of query timeout period.

4. Do not close the connection anymore when rpc timeout occurs.
5. Change some log level from info to debug to simplify the fe.log content.

NOTICE:
1. Change the definition of execPlanFragment rpc, must first upgrade BE.
3. Remove FE config `remote_fragment_exec_timeout_ms`
2022-06-03 15:47:40 +08:00
cacad64d2c [fix] Error when compiling under ARM architecture machine, fixed a loop (#9948) 2022-06-03 08:00:55 +08:00
Pxl
c0ad1be1bd [Enhancement][Chore] remove breakpad and unused variable (#9937) 2022-06-02 20:52:17 +08:00
c426c2e4b1 [Vectorized-Load] Support vectorized load table with materialized view (#9923)
* [Vectorized-Load] Support vectorized load table with materialized view

* fix ut

Co-authored-by: lihaopeng <lihaopeng@baidu.com>
2022-06-02 14:59:01 +08:00
4ea5782838 [fix](Function) fix to_bitmap to return always not nullable (#9859) 2022-06-02 10:37:45 +08:00
e896fffd76 [Vectorized][Function] fix bitmap_intersect get wrong result (#9907) 2022-06-01 23:51:52 +08:00
47dfdd8e09 [fix](storage) Disable compaction before schema change is actually executed(#9032) (#9065)
As in issue, the combination and schema change at the same time may lead to version intersection.
Describe the overview of changes.
1. Do not do compaction before schema change is actually executed.
2. Set tablet as bad when it has version intersection.
3. Do not do schema change when it can not find appropriate versions to delete in new tablet.
4. Do not change rowsets after compaction if the rowsets of the tablet has changed.
2022-06-01 23:29:18 +08:00
Pxl
ac08c7ac91 [fix](vectorized) fix vcast expr input wrong row number (#9520) 2022-06-01 15:19:31 +08:00
632f7a3d3d [Feature] add weekday function on vectorized engine (#9901) 2022-06-01 14:47:37 +08:00
00719db3a2 [bugfix]handle ComlumnDictionary in evaluate_and and evaluate_or (#9818)
* handle ColumnDictory in evaluate_or

We need to handle ComlumnDictory in evaluate_or, otherwise delete handler
would triger a core dump.

* handle ComlumnDictionary in evaluate_and

Because there is only one difference between evaluate_and and
evaluate_or, that is or and delete, I merge two macros into one.

Delete handlers also trigger evaluate_and, i am not sure if column
dictionary would be used in evaluate_and.

* clang format

* fix short circut for evaluate_and and evaluate_or

* clang format
2022-06-01 08:10:43 +08:00
35f99faa0a [Bug][Vectorized] fix core dump on vcase_expr::close (#9893)
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
2022-06-01 08:05:09 +08:00
0376ca17f3 [Enhancement] Remove minidump (#9894) 2022-06-01 08:04:24 +08:00
f3193c5ea3 [improvement]opt column_dictinary range filter (#9881)
* opt column_dictinary range filter

* fomart
2022-05-31 22:30:05 +08:00
54e9d49718 [Bug][Vectorized] Fix call nvl function core dump (#9883)
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
2022-05-31 22:18:38 +08:00
Pxl
d34d631519 [bugfix]fix TableFunctionNode memory leak (#9853) 2022-05-31 19:20:22 +08:00
c8d303a82c [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate 2022-05-31 19:12:42 +08:00
Pxl
fa50b63cee fix core dump on vcase_expr::close (#9875) 2022-05-31 15:45:39 +08:00
0cba6b7d95 [Bug][Fix] One Rowset have same key output in unique table (#9858)
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
2022-05-31 12:29:16 +08:00
7199102d7c [Opt][VecLoad] Opt the vec stream load performance (#9772)
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
2022-05-31 11:53:32 +08:00
7b55d4cb88 [BUG] return NULL for invalid date value (#9862) 2022-05-30 21:35:41 +08:00
85f525e991 [Bugfix(Vec)] Close result_sink properly (#9849)
Close result_sink properly so that error code is reported and
expr_context is always closed.
2022-05-30 19:03:33 +08:00
f377c26bf7 [refactor][be] Optimize headers (#9708) 2022-05-30 16:12:10 +08:00
4af2493c42 [Improvement] optimize scannode concurrency query performance in vectorized engine. (#9792) 2022-05-30 16:04:40 +08:00
7b98dd438d [feature](function) Add nvl function (#9726) 2022-05-30 09:43:00 +08:00
0683181fef [API changed](parser) Remove merge join syntax (#9795)
Remove merge join sql and merge join node
2022-05-30 09:04:21 +08:00
a96b41db7a [Improvement] Simplify expressions for _vconjunct_ctx_ptr (#9816) 2022-05-29 23:05:21 +08:00
63aab5ee5d [Bugfix(Vec)] Fix some memory leak issues (#9824) 2022-05-29 23:04:11 +08:00
1aeb16d153 [improvement](load) reduce useless err_msg format in VOlapTableSink send (#9531) 2022-05-29 16:02:57 +08:00
9fe3827239 [fix](ut) fix BE ut (#9831)
introduced from #8923, the github checks has some problem that failed to check BE ut in #8923
2022-05-29 12:25:41 +08:00
Pxl
f33ef32d92 [Bug] [Bitmap] change to_bitmap to always_not_nullable (#9716) 2022-05-28 17:33:55 +08:00
4d1e926b6c [feature][config] introduce a new BE config storage_page_cache_shard_size (#9821)
Co-authored-by: gaodayue <gaodayue@bytedance.com>
2022-05-28 10:17:09 +08:00