In Memo.copyIn( plan, group1, isRewrite), one branch is that the plan is already recorded in Memo, and owned by group 'group2'. In such case, 'group1' should be merged with 'group2', because they are equivalent.
After merge, the upper level of 'group1', saying 'p1 = group1.getLogicalExpression().getOwnerGroup()' of 'group1', and that of 'group2', saying 'p2', are equivalent. We need to merge 'p1' and 'p2'. And this process is recursive.
Hash join node adds three new attributes.
The following will take an SQL as an example to illustrate the meaning of these three attributes
```
select t1. a from t1 left join t2 on t1. a=t2. b;
```
1. vOutputTupleDesc：Tuple2(a'')
2. vIntermediateTupleDescList: Tuple1(a', b'<nullable>)
2. vSrcToOutputSMap: <Tuple1(a'), Tuple2(a'')>
The slot in intermediatetuple corresponds to the slot in output tuple one by one through the expr calculation of the left child in vsrctooutputsmap.
This code mainly merges the contents of two PRs:
1. [fix](vectorized) Support outer join for vectorized exec engine (https://github.com/apache/doris/pull/10323)
2. [Fix](Join) Fix the bug of outer join function under vectorization #9954
The following is the specific description of the first PR
In a vectorized scenario, the query plan will generate a new tuple for the join node.
This tuple mainly describes the output schema of the join node.
Adding this tuple mainly solves the problem that the input schema of the join node is different from the output schema.
For example:
1. The case where the null side column caused by outer join is converted to nullable.
2. The projection of the outer tuple.
The following is the specific description of the second PR
This pr mainly fixes the following problems:
1. Solve the query combined with inline view and outer join. After adding a tuple to the join operator, the position of the `tupleisnull` function is inconsistent with the row storage. Currently the vectorized `tupleisnull` will be calculated in the HashJoinNode.computeOutputTuple() function.
2. Column nullable property error problem. At present, once the outer join occurs, the column on the null-side side will be planned to be nullable in the semantic parsing stage.
For example：
```
select * from (select a as k1 from test) tmp right join b on tmp.k1=b.k1
```
At this time, the nullable property of column k1 in the `tmp` inline view should be true.
In the vectorized code, the virtual `tableRef` of tmp will be used in constructing the output tuple of HashJoinNode (specifically, the function HashJoinNode.computeOutputTuple()). So the **correctness** of the column nullable property of this tableRef is very important.
In the above case, since the tmp table needs to perform a right join with the b table, as a null-side tmp side, it is necessary to change the column attributes involved in the tmp table to nullable.
In non-vectorized code, since the virtual tableRef tmp is not used at all, it uses the `TupleIsNull` function in `outputsmp` to ensure data correctness.
That is to say, the a column of the original table test is still non-null, and it does not affect the correctness of the result.
The vectorized nullable attribute requirements are very strict.
Outer join will change the nullable attribute of the join column, thereby changing the nullable attribute of the column in the upper operator layer by layer.
Since FE has no mechanism to modify the nullable attribute in the upper operator tuple layer by layer after the analyzer.
So at present, we can only preset the attributes before the lower join as nullable in the analyzer stage in advance, so as to avoid the problem.
(At the same time, be also wrote some evasive code in order to deal with the problem of null to non-null.)
Co-authored-by: EmmyMiao87
Co-authored-by: HappenLee
Co-authored-by: morrySnow
Co-authored-by: EmmyMiao87 <522274284@qq.com>
This is an example of s3 hms_catalog:
```sql
CREATE CATALOG hms_catalog properties(
"type" = "hms",
"hive.metastore.uris"="thrift://localhost:9083",
"AWS_ACCESS_KEY" = "your access key",
"AWS_SECRET_KEY"="your secret key",
"AWS_ENDPOINT"="s3 endpoint",
"AWS_REGION"="s3-region",
"fs.s3a.paging.maximum"="1000");
```
All these params are necessary;
1.This pr is used for adding the supported sub-type for array which has been modified in #9916
2.add regression test for the supported sub-type
Co-authored-by: hucheng01 <hucheng01@baidu.com>
# first: Add two expr rewrite rule:
1. remove duplicate expr
a = 1 and a = 1 -> a = 1
2. extract common expr
(a or b) and (a or c) -> a or (b and c)
# second: Add some plan rewrite rule of rewriting expr of operator
1. NormalizeExpressionOfPlan contains normalize expr rewrite rule. Using these normalizerule rewrite LogicalFilter、LogicalAggravate，LogicalProject，LogicalJoin exprs
2. OptimizeExpressionOfPlan contains optimize expr rewrite rule. Using these optimize rule rewrite LogicalFilter、LogicalAggravate，LogicalProject，LogicalJoin exprs
review and add all missing equals and hashCode function to Expression and its sub class.
Alias
Arithmetic
BoundFunction
CompoundPredicate
Not
UnboundFunction
UnboundSlot
UnboundStar
support case when for TPC-H
for example:
CASE [expression] WHEN [value] THEN [expression] ... ELSE [expression] END
or
CASE WHEN [predicate] THEN [expression] ... ELSE [expression] END
GroupExpression.getParent() returns the group which contains this expr. This name is missleading especially in tree structures.
So we change the name to getOwnerGroup.
Try to eliminate cross join via finding join conditions in filters and changing the join orders.
For example:
-- input:
SELECT * FROM t1, t2, t3 WHERE t1.id=t3.id AND t2.id=t3.id
-- output:
SELECT * FROM t1 JOIN t3 ON t1.id=t3.id JOIN t2 ON t2.id=t3.id
This feature is controlled by session variable enable_nereids_reorder_to_eliminate_cross_join with true by default.
Simplify usage of Memo and rewrite rule application.
Before this PR, if we want to apply a rewrite rule to a plan, the code is like the below:
Memo memo = new Memo();
memo.initialize(root);
PlannerContext plannerContext = new PlannerContext(memo, new ConnectContext());
JobContext jobContext = new JobContext(plannerContext, new PhysicalProperties(), 0);
RewriteTopDownJob rewriteTopDownJob = new RewriteTopDownJob(memo.getRoot(),
ImmutableList.of(new AggregateDisassemble().build()), jobContext);
plannerContext.pushJob(rewriteTopDownJob);
plannerContext.getJobScheduler().executeJobPool(plannerContext);
Plan after = memo.copyOut();
After this PR, we could use chain style calling:
new Memo(plan)
.newPlannerContext(connectContext)
.setDefaultJobContext()
.topDownRewrite(new AggregateDisassemble())
.getMemo()
.copyOut();
Rename the session variable enable_nereids to enable_nereids_planner to make it more meaningful.
During the analysis of BinaryPredicate, it will generate a CastExpr if the slot implicitly in the below case:
SELECT * FROM t1 WHERE t1.col1 = '1';
col1 is integer column.
This will prevent the binary predicate from pushing down to OlapScan which would impact the performance.
This pull request includes some implementations of the statistics(https://github.com/apache/incubator-doris/issues/6370), it will not affect any existing code and users will not be able to create statistics job.
Now only MetaStatisticsTask that directly collects statistics by reading FE meta is implemented. SQLStatisticsTask is still being implemented, it needs to query BE through FE.
The following is the function implemented by this pr:
1. Support statistics collection for partitioned and non-partitioned tables. For partitioned tables, the collection of statistics for the specified partition is implemented.
2. When the task is divided, it is divided according to the partition table and the non-partition table. The most fine-grained is to the tablet level. A matetask collects as many statistics as possible.
3. Add partition statistics (Table -> Partition -> Column). For example, the size of the table, the number of rows, the size of the partition, the number of rows, the maximum and minimum values of the columns, etc.
4. Display and modify partition-level statistics.
…
Add hashCode(), equals() for operator.
Add basic UT for them(need more detail test).
**future ticket**: add hashCode(), equals() and UT for `Expression`
Compilation error after merging due to Literal refactoring.
Compilation failure:
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/Substring.java:[40,38] org.apache.doris.nereids.trees.expressions.Literal is abstract; cannot be instantiated
