Commit Graph

4831 Commits

Author SHA1 Message Date
dc874709d7 [feature-wip](array-type) support array<decimal128> in mysql_result_writer (#9998) 2022-06-09 15:15:26 +08:00
6fab1cbf3c [feature-wip](array-type) Add array functions size and cardinality (#9921)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-06-09 15:03:03 +08:00
9bc82542fd [test] add bitmap index regression test (#10008) 2022-06-09 12:19:07 +08:00
050cbba6e5 [fix][hudi] use lowerCase to get hudi fileFormatType (#9873)
use lowerCase of inputFormatName to get hudi fileFormatType
2022-06-09 12:13:02 +08:00
19bc14cf8d [feature-wip](array-type) Add array type support for vectorized parquet-orc scanner (#9856)
Only support one level array now.
for example:
- nullable(array(nullable(tinyint))) is **support**.
- nullable(array(nullable(array(xx))) is **not support**.
2022-06-09 12:11:47 +08:00
bf8b4fb2d3 [Bugfix] be crash when executing sql contains bitmap_intersect function (#9910)
* fix bitmap serialize bug

* add regression test for bitmap seralize bugfix

* add missing regression test out file

* fix reggresion test failed issue
2022-06-09 08:45:46 +08:00
e5ed259da6 [fix](regression-test) fix the comparison errors caused by inconsistent accuracy (#9931) 2022-06-09 00:35:22 +08:00
449bfe10d1 fix: fix a thread safe problem in LoadAction.java (#9955) 2022-06-09 00:34:07 +08:00
688f8f6e90 [test] Fix date function regression test case. (#9966) 2022-06-09 00:33:35 +08:00
342ab52270 [fix] Fix type description in PrimitiveType (#9985) 2022-06-09 00:30:32 +08:00
99fb830023 [feature] datetime column type support auto-initialized with default … (#9972) 2022-06-09 00:28:03 +08:00
9c52b4a508 [enhance] improve dict in-predicate evaluate (#10009) 2022-06-09 00:25:30 +08:00
5f56e17ef2 [feature-wip](multi-catalog)(step2) Introduce Internal Data Source (#9953) 2022-06-08 22:02:22 +08:00
2d39cffa5c [doc](website)Add Doris new official website code and documents (#9977)
In order to cooperate with Doris's successful graduation from Apache, the Doris official website also needs a new look
and more powerful feature, so we decided to redesign the Doris official website.
The code and documents of the new official website are included in this PR.

Since the new website is completely rewritten, the content and structure of the project are different from the previous one. 
In particular, the directory structure of documents has changed, and the number of documents is large, so the number of 
files in this PR is very large.

In the old website,all English documents are in the en/ directory, and Chinese documents in the zh-CN/ directory,
but in the new website,the documents are split into multiple directories according to the nav.
The document's directory structure changes as follows:
```
docs (old website)
|   |—— .vuepress (library)
|   |—— en
|   |   |—— admin-manual 
│   │   |—— advanced
|   |   |—— article
|   |   |—— benchmark
|   |   |—— case-user
|   |   |—— community
|   |   |—— data-operate
|   |   |—— data-table
|   |   |—— design
|   |   |—— developer-guide
|   |   |—— downloads
|   |   |—— ecosystem
|   |   |—— faq
|   |   |—— get-starting
|   |   |—— install
|   |   |—— sql-manual
|   |   |—— summary
|   |   |___ README.md
|   |—— zh-CN
...

docs (new website)
|   |—— .vuepress (library)
|   |—— en
|   |   |—— community (unchanged, community nav)
│   │   |—— developer (new directory, developer nav)
│   │   |   |—— design (moved from en/design)
│   │   |   |__ developer-guide (moved from en/developer-guide)
|   |   |—— docs (new directory, all children directories moved from en/, document nav)
│   │   |   |—— admin-manual 
│   │   |   |—— advanced
│   │   |   |—— benchmark
│   │   |   |—— data-operate
│   │   |   |—— data-table
│   │   |   |—— ecosystem
│   │   |   |—— faq
│   │   |   |—— get-starting
│   │   |   |—— install
│   │   |   |—— sql-manual
│   │   |   |—— summary
|   |   |—— downloads (unchanged, downloads nav)
|   |   |—— userCase (moved from en/case-user, user nav)
|   |   |___ README.md
|   |—— zh-CN
...
```
2022-06-08 17:45:12 +08:00
d9bbf67b9e [DefaultConfigChange]enable query vectorization and storage vectorization and storage low cardinality optimization by default (#9848)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2022-06-08 15:29:43 +08:00
94089b9192 [Refactor] Use file factory to replace create file reader/writer (#9505)
1. Simplify code logic and improve abstraction
2. Fix the mem leak of raw pointer

Co-authored-by: lihaopeng <lihaopeng@baidu.com>
2022-06-08 15:07:39 +08:00
2ed523440f [fix](planner) passthrough child in SetOperationNode is wrong when enable vector engine (#9991)
In SetOperationNode we do passthrough, if we child output is same with itself output.
In method isChildPassthrough we only consider memory layout.
When we use vectorized engine, we need to use SlotDesc offset in TupleDesc instead of
memory layout to check whether pass-through can be performed
2022-06-08 14:12:04 +08:00
fc9afda97a [enhancement][diagnostics] Add a diagnostic: detect unused includes (#9117) 2022-06-08 11:52:48 +08:00
b15b1a9f07 [tools](tpc-h) Modify tpch tools (#9982)
1. Modify the create table stmt.
2. Modify part of queries.
2022-06-08 11:21:57 +08:00
35c3e4e33c [Bug] runtime filter is not used as expected (#10001)
* [Bug] runtime filter is not used as expected

* update
2022-06-08 11:10:39 +08:00
dcdfc5b32a [fix](coordinator) fix bug that unable to generate query profile (#10002)
This bug was introduced from #9720
2022-06-08 10:59:15 +08:00
b6c45327ed [revert](test) delete bitmap_index test case (#9999)
This test may cause endless loop
2022-06-08 08:56:23 +08:00
e97d835ba7 [feature](statistics) Statistics derivation.Step 2:OtherNode implemen… (#9458)
closed #9644

Second step of statistics derivation: implementation of nodes other than scan_node.
The statistical information derivation interface of all nodes is uniformly placed in DeriveFactory.java.
Added one-sided to verify the derivation is correct.

Statistics derivation for each node is placed in its own *StatsDerive.java
detailed design: https://docs.google.com/document/d/1u1L6XhyzKShoyYRwFQ6kE1rnvY2iFwauwg289au5Qq0/edit
2022-06-07 21:10:28 +08:00
Pxl
f2aa5f32b8 [Feature] [Vectorized] Some pre-refactorings or interface additions for schema change (#9811)
Some pre-refactorings or interface additions for schema change
2022-06-07 15:04:57 +08:00
d588e99b8b [fix][mem tracker] Fix logout load task mem tracker dcheck fail (#9943)
* fix tracker 0602

* fix format
2022-06-07 11:31:49 +08:00
0fa1615147 [fix](fe-ut) Fix FE ut when enable vectorized engine (#9958)
Some node name in query explain result will be changes.
eg:
Aggregate -> VAggregate
2022-06-07 09:13:47 +08:00
856b421086 [feature](priv) Support grant node_priv to other user. (#9951)
Currently, only the root user has node_priv privileges.
That is, only the root user can operate the addition and deletion of nodes.

In the original design of Doris, there is an Operator role. This role can have node_priv for node operations.

This PR supports assigning node_priv to users other than root.
However, only users who have both grant_priv and node_priv can assign node_priv to other users.
This ensures that only the root user has this permission, and users who are given node_priv
cannot continue to expand this permission outward.
2022-06-06 11:04:20 +08:00
24ad11af6a [deps] upgrade fabric8 k8s client to compitable new k8s cluster (#9933) 2022-06-06 10:00:36 +08:00
e34b2e2224 [improvement] set default jvm memory to 8G for FE (#9683) 2022-06-06 08:59:20 +08:00
49d4798276 [fix](function) fix bug in time_round function (#9712) 2022-06-06 08:58:22 +08:00
c18f7a31f1 remove redundant this (#9878)
Co-authored-by: vishalsingh <2018uec1001@gmail.com>
2022-06-05 13:09:14 +08:00
40b720473c [test] Add regression test case for Table function (#9947) 2022-06-04 13:32:09 +08:00
26ea51da00 [test]Add regression test method for agg function (#9929)
* Add regression test method for regression function

* Add regression test method for regression function

* Add regression test case for math function

* delete math function

Co-authored-by: manyi <fop@freeoneplus.com>
2022-06-04 13:31:48 +08:00
79620f6fa2 [Improvement] change the condition of vectorized compaction (#9950) 2022-06-04 12:29:23 +08:00
3743f19369 [feature] support convert alpha rowset (#9890)
Add alpha rowset to beta rowset convert to convert rowset automatically. We will remove alpha rowset's code after 1.1.
2022-06-04 12:29:03 +08:00
f49284036e [Enhancement] Refactor functions in int_exp by templates (#9939) 2022-06-04 11:53:31 +08:00
da33a48f39 [refactor](policy) Refactor the hierarchy of Policy. (#9786)
The RowPolicy extends Policy
2022-06-04 11:29:09 +08:00
3031919e8f [fix] (planner) slot nullable does not set correctly when plan outer join with inline view (#9927)
- set inline view's slot descriptor to nullable in register column ref
- propagate slot nullable when generate inline view's query node in SingleNodePlanner
2022-06-03 17:50:10 +08:00
937491098e [fix] fix grammar of ADMIN SHOW TABLET STORAGE FORMAT stmt (#9938) 2022-06-03 17:49:34 +08:00
c996334ad1 [improvement] Optimize send fragment logic to reduce send fragment timeout error (#9720)
This CL mainly changes:
1. Reducing the rpc timeout problem caused by rpc waiting for the worker thread of brpc.
    1. Merge multiple fragment instances on the same BE to send requests to reduce the number of send fragment rpcs
    2. If fragments size >= 3, use 2 phase RPC: one is to send all fragments, two is to start these fragments. So that there
         will be at most 2 RPC for each query on one BE.

3. Set the timeout of send fragment rpc to the query timeout to ensure the consistency of users' expectation of query timeout period.

4. Do not close the connection anymore when rpc timeout occurs.
5. Change some log level from info to debug to simplify the fe.log content.

NOTICE:
1. Change the definition of execPlanFragment rpc, must first upgrade BE.
3. Remove FE config `remote_fragment_exec_timeout_ms`
2022-06-03 15:47:40 +08:00
cacad64d2c [fix] Error when compiling under ARM architecture machine, fixed a loop (#9948) 2022-06-03 08:00:55 +08:00
67fa1fcf2a [fix] fix invalid SQL rewrite for field in materialized view (#9877) 2022-06-02 23:43:13 +08:00
3241cc2bfd [doc][fix]English document title case (#9935) 2022-06-02 20:52:58 +08:00
Pxl
c0ad1be1bd [Enhancement][Chore] remove breakpad and unused variable (#9937) 2022-06-02 20:52:17 +08:00
c426c2e4b1 [Vectorized-Load] Support vectorized load table with materialized view (#9923)
* [Vectorized-Load] Support vectorized load table with materialized view

* fix ut

Co-authored-by: lihaopeng <lihaopeng@baidu.com>
2022-06-02 14:59:01 +08:00
df3406da30 [test]add string sql functions test case (#9904)
* add string sql functions test case

* add like and regexp function case vectorized true

* Cancellation of Chinese language Test

* Cancel the Chinese test for the second time
2022-06-02 14:53:07 +08:00
0be1b0d242 [regressiontest]add json function test case (#9917) 2022-06-02 14:52:39 +08:00
7792aebd21 [test]add conditional function test case (#9913)
* add conditional function test case

* Update test_conditional_function.groovy

Delete useless code

* optimize the code
2022-06-02 14:52:03 +08:00
2ad38e6ab2 [regressiontest]add hash function test case (#9918) 2022-06-02 14:51:25 +08:00
f681c0891f [regression test]add encryption function test case (#9920) 2022-06-02 14:47:32 +08:00