Commit Graph

1468 Commits

Author SHA1 Message Date
b15e2ddeaa [fix](Nereids): fix and enable stats derive job (#11755)
fix and enable statistics derive job
Add mock for statistics in computeScan().
2022-08-18 21:26:35 +08:00
b9dcb60172 [Planner](fix)Fix unexpected index out of bound exception (#11819) 2022-08-18 15:52:54 +08:00
066bc7693e [fix](orderby)disallow hll and bitmap data type in order by list (#11837) 2022-08-18 14:50:25 +08:00
b8a33d2629 [Improvement](load) turn enable_vectorized_load on by default (#11833) 2022-08-18 14:43:09 +08:00
0637c339b1 [fix](array-type) support to insert the largeint in array (#11868)
Co-authored-by: hucheng01 <hucheng01@baidu.com>
2022-08-18 14:41:07 +08:00
4c3f72d019 [improvement](meta) sort result by tablename when show table status like 'show data' (#11885) 2022-08-18 14:23:45 +08:00
3eeaa8e65b [typo](fix) Fix spelling errors in comments (#11810) 2022-08-18 13:55:41 +08:00
0903dd61f3 [Enhancement](Planner) Improve error message when columns order and keys orders don't match. (#11724)
When creating table like this:
```
CREATE TABLE `test`.`test_key_order` (
  `k1` tinyint(4) NULL COMMENT "",
  `k2` smallint(6) NULL COMMENT "",
  `k3` int(11) NULL COMMENT "",
  `v1` double MAX NULL COMMENT "",
  `v2` float SUM NULL COMMENT ""
) ENGINE=OLAP
AGGREGATE KEY(`k1`, `k3`, `k2`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`k1`) BUCKETS 5
PROPERTIES (
"replication_num" = "1"
);
```

The error message before is:
```
Key columns should be a ordered prefix of the schema.
```

With this PR, the error message is:
```
Key columns should be a ordered prefix of the schema. KeyColumns[1] (starts from zero) is k3, but corresponding column is k2 in the previous columns declaration.
```
2022-08-18 13:28:54 +08:00
e1a1a04c2f [Enhancement](Doe) Be query es use fe generate dsl. (#11840) 2022-08-18 10:31:17 +08:00
6c66bdbf30 [fix](orderby)remove useless null literal in order by (#11821) 2022-08-18 10:10:25 +08:00
582be130dd [Feature] (ODBC) support read/write emoji of utf16 via odbc table (#11863)
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
2022-08-18 09:09:02 +08:00
4cdf9f2a23 [Enhancement](Nereids) Refine nereids parser. (#11839)
1. Use ParseException in nereids parser.
2. Add check utils in the parser test.
3. Distinguish matchesFromRoot and matches when checking plans.
2022-08-17 20:17:26 +08:00
4a4d3b273d fix reorder error (#11854)
join reorder throw unexpected exception when join type is not cross and inner
2022-08-17 17:26:24 +08:00
790a1d681f [Bug](external iceberg table)Fix iceberg on ha-hdfs unknown hostname bug. #11844 2022-08-17 16:21:30 +08:00
98243e99ae [feature-wip](unique-key-merge-on-write) unique key table with MOW supports delete sign column (#11672) 2022-08-17 15:12:11 +08:00
ba3e0b3f96 [feature](compaction) allow to set disable_auto_compaction for tables (#11743) 2022-08-17 11:05:47 +08:00
12c4d1f4dd [feature-wip](unique-key-merge-on-write) unique key table with MOW supports sequence column (#11808) 2022-08-17 10:56:14 +08:00
3e13b7d2c2 [Bugfix](light-shema-change) fix _finish_clone dead lock (#11823)
In engine_clone_task.cpp, it use tablet->tablet_schema() to create rowset, but in the method, it need a lock that already locked in engine_clone_task.cpp:514. It use cloned_tablet_meta->tablet_schema() originally, but modified in #11131. It need to revert to use cloned_tablet_meta->tablet_schema().
2022-08-17 09:10:08 +08:00
a07e153419 [Feature](nereids)support view and nested view (#11589)
support view in query
and add a rewrite rule: merge consecutive projects.
the rule can merge relative consecutive projects to one project to improve efficiency
2022-08-16 19:24:01 +08:00
f2292a3b1d [Enhancement](array-type) enable_array_type flag update (#11785)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-08-16 14:41:57 +08:00
cf45151a66 [Fix](audit) do not use duplicate query id in fe.audit.log (#11746)
The original logic in ConnectProcessor.java might result in duplicate query id for different query statement in fe.audit.log as follows.
2022-08-16 14:37:16 +08:00
56574b5948 [refactor] rename data source to catalog (#11243)
In #10702, the origin Catalog class has been renamed to Env.
Now we can rename the datasource to catalog.
2022-08-16 11:45:49 +08:00
f3f1bbc48c [fix](agg)disallow group by bitmap or hll data type (#11782)
* [fix](agg)disallow group by bitmap or hll data type
2022-08-16 09:25:02 +08:00
d2d4423c88 [feature](schema change) Light schema change support rollup (#11494)
1. Move max colUniqueId from OlapTable to IndexMeta.
2. Add updateSlotUniqueId.
2022-08-15 21:39:27 +08:00
5104982614 [enhancement](tracing) append the profile counter to trace. (#11458)
1. append the profile counter and infos to span attributes.
2. output traceid to audit log.
2022-08-15 21:36:38 +08:00
8f98357c0b [fix](array-type) disable cast function to array type on origin exec engine. (#11602)
This commit disable cast to array type on origin exec engine, except cast varchar to array type.
2022-08-15 21:30:56 +08:00
fef37990a3 [fix](array-type) Fix incorrect in function-set for array type (#11585)
There is some wrong logic in FunctionSet.java and it may causes potential risks for array functions invoke.
2022-08-15 21:29:57 +08:00
910d51c76f [fix](update) Fix where clause is not reanalyzed after rewrite (#11723) 2022-08-15 13:24:57 +08:00
1e6b8cd1a9 [feature](nereids): SimplifyCastRule (#11630)
Remove redundant cast like
```
cast(1 as int) -> 1
```
2022-08-15 12:41:36 +08:00
66116b78d1 fix multJoin bug (#11772)
In MultiJoin transform, if there is no condition, we should generate cross join instead of inner join
2022-08-15 11:59:09 +08:00
854b4b1b47 [fix](storage-policy) fix bug that missing field when refreshing storage policy (#11580)
1. Change all required fields to optional
    Although they all "required", but it not recommended to use `required`, because it is hard to modify in future.
2. Fix a missing field bug
2022-08-12 20:11:54 +08:00
ed47a3bb6d [enhancement](Nereids)refactor sort plan in nereids (#11673)
1. rename PhysicalHeapSort to PhysicalQuickSort
2. add LogicalTopN and PhysicalTopN
3. add implementation rule for LogicalTopN
4. add a interface Sort for both logical and physical sort
5. add a interface TopN for both logical and physical top-n
6. add a AbstractPhysicalSort as super class of PhysicalQuickSort and PhysicalTopN
2022-08-12 17:08:23 +08:00
887de4b465 [fix](Nereids)memo copy in cannot rewrite current plan with its child (#11717)
When we do write current plan with its child. Currently, nereids memo do nothing indeed.
This PR fix this situation by add a predicate in rewrite that only replaced target group and set newGroupExpressionGenerated to false when target is null or target is same with existed group expression's group.

here we need to handle one situation that original target is not the same with
existedGroupExpression.getOwnerGroup(). In this case, if we change target to
existedGroupExpression.getOwnerGroup(), we could not rewrite plan as we expected and the plan
will not be changed anymore.
Think below example:
We have a plan like this:
```
Original (Group 2 is root):
Group2: Project(outside)
Group1: |---Project(inside)
Group0:     |---UnboundRelation

and we want to rewrite group 2 by Project(inside, GroupPlan(group 0))

After rewriting we should get (Group 2 is root):
Group2: Project(inside)
Group0: |---UnboundRelation
Group1: Project(inside)
```

After rewriting, Group 1's GroupExpression is not in GroupExpressionsMap anymore and Group 1 is unreachable.
Merge Group 1 into Group 2 is better, but in consideration of there is others way to let a Group take into
unreachable. There's no need to complicate to add a merge step. Instead, we need to have a clear step to
remove unreachable groups and GroupExpressions after rewrite.
2022-08-12 16:20:46 +08:00
ec4347ad39 [enhancement](Nereids) support StatementContext, SET_VAR, and Plan pre/post processor (#11654)
1. add StatementContext, and PlannerContext is renamed to CascadsContext. CascadsContext belong to a StatementContext, and StatementContext belong to a ConnectionContext, and the lifecycle increases in turn. StatementContext can wrap some statement's lifecycle-related state, such as ExpressionId, TableLock. MemoTestUtil can simplify create a CascadesContext and Memo for test.
2. add PlanPreprocessor to process parsed logical plan before copy into memo. and add a PlanPostprocessor to process physical plan after copy out from memo.
3. utilize PlanPreprocessor to process SET_VAR hint, the class is EliminateLogicalSelectHint
4. pass the limit clause in regression test case, in set_var.groovy
2022-08-12 14:49:11 +08:00
a3ae1494ef [Enhancement](admin show replica distribution) size percent should be 0.00% when size is zero #11711
Co-authored-by: wuhangze <wuhangze@jd.com>
2022-08-12 13:58:56 +08:00
800bc100af [feature-wip](unique-key-merge-on-write) show enable_unique_key_merge_on_write property when get ddl (#11664)
* [feature-wip](unique-key-merge-on-write) show enable_unique_key_merge_on_write property when get ddl

* update
2022-08-12 12:14:04 +08:00
7d97aa194b [feature-wip](datev2) Support to use datev2 as partition column (#11618) 2022-08-12 11:54:01 +08:00
6c6328fc6d [fix](join)fix outer join bug when a subquery as nullable side #11700 2022-08-12 11:50:15 +08:00
32e451e6ad [minor](recycle) add log to observe bug (#11687) 2022-08-12 11:40:20 +08:00
Pxl
f5fe622a1b [Bug](materialized view) fix create materialized view fail
1. remove referenced_column(seems unused now).
2. fix mv slot ref id wrong.
3. add type check for hll_hash.
4. enable non-nullable column change to nullable column.
2022-08-12 09:49:16 +08:00
5d66839035 [feature-wip](unique-key-merge-on-write) push down runtime filter on unique key with merge on write table (#11695) 2022-08-11 22:50:13 +08:00
2d5ffac590 [fix](optimization) InferFiltersRule bug: a self inner join on a view, which contains where clause, will cause mis-inference. (#11566) 2022-08-11 17:13:26 +08:00
9214346603 [Bug] fix cancel fragment bug (#11683)
Co-authored-by: chenjie <chenjie@cecdat.com>
2022-08-11 16:48:18 +08:00
aa24c8f973 [Feature](multi-catalog) Support refresh catalog metadata (#11656) 2022-08-11 15:14:05 +08:00
7e828ad85b [Enhancement](profile format) Print fragment and instance in a better format (#11671) 2022-08-11 14:05:41 +08:00
180cc35815 [Feature](nereids) support sub-query and alias in FromClause (#11035)
Support sub-query and alias for TPC-H,for example:
select * from (select * from (T1) A join T2 as B on T1.id = T2.id) T;
2022-08-11 12:42:19 +08:00
d831322806 [refactor](planner): make Table abstract. (#11642) 2022-08-11 12:07:25 +08:00
56ce8c0c5c [feature](http) add api for showing current queries and kill query (#11657) 2022-08-11 10:32:46 +08:00
04d26ddf22 [feature-wip](multi-catalog)Support use catalog.db and show databases from catalog stmt (#11338)
Support use catalog.db and show databases from catalog stmt.
2022-08-11 09:50:32 +08:00
02a3f21b65 [fix](analyzer) InferFilterRule bug: equations in on clause of outer/anti join are not inferable. (#11515) 2022-08-11 09:36:43 +08:00