We have added logical project before, but to actually finish the prune to reduce the data IO, we need to add related supports in translator and BE.
This PR:
- add projections on each ExecNode in BE
- translate PhysicalProject into projections on PlanNode in FE
- do column prune on ScanNode in FE
Co-authored-by: HappenLee <happenlee@hotmail.com>
We can skip aggregate on replace column, otherwise it would generate
wrong result. e.g. a row in UNIQUE is deleted by delte_sign_column,
then it would be returned.
Array column should not be distributed key or group by or aggregate key, we should forbid it before create table.
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
Add one expression rewrite rule:
rewrite InPredicate to an EqualTo Expression, if there exists exactly one element in InPredicate Options.
Examples:
1. where A in (x) ==> where A = x
2. where A not in (x) ==> where not A = x
In BE, There is an implicit convention that HashJoinNode's left child's output Slot must before right child's output slot in intermediateTuple.
However, after we do commute rule on join plan in Nereids, this convention will be broken and cause core dump in BE.
There are two way to fix this problem:
1. add a project on join after we do commute
2. reorder output of join node when we do translate
Since we cannot translate project yet because BE projection support is on going(#11842). So we use second way to fix it now. After the project translation could work correctly, we should use the first way to fix it.
Problem:
1. `enable_array_type` is masterOnly;
2. dynamic open config only affect FE MASTER
`admin set frontend config("enable_array_type"="true");`
3. query in FE FOLLOWER will fail, because of `enable_array_type` is false in FE FOLLOWER
`select * from table_with_array `
Solution:
Only check `enable_array_type` while creating new tables with array column.
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
1. Rewrite Filter(Project) to Project(Filter) to make sure when do partition prune the tree looks like this: Project(Filter(OlapScan)).
2. Enable the MergeConsecutiveProject MergeConsecutiveFilter rules.
3. prune range partition just like what Legacy Planner do.
This PR fix three problem in Nereids.
- Add selected index, partition and tablet Info in LogicalOlapScan
- with JoinReorderContext in new LogicalJoin
- fix compute data size when no column size info in StatsCaculator
* table function node enhancement
* also avoid copy for non-vec table function node
* fix table function node output slots calculation while lateral view involves subquery
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
This API can help user to get all create table statement of a given SQL.
So that it will be easier to run the case in other Doris cluster.
See document for details
Currently, the join on conditions are denoted by Join.conditions as one expression.
It is not convenient to get equal-predicates for hash table.
This pr add a rule FindHashConditionForJoin to split on-conditions into two parts:
- hashJoinPredicates
- otherJoinConditions
And store them as two attributes in LogicalJoin and PhysicalJoin.
JoinReorder rule is impacted by this change, since the way we get the equal conditions between join children are changed.
In this pr, we changed the output of multiJoin slightly. Equations like A=1 are moved from join condition to upper filter.
For more detail, refer to fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/ssb/SSBJoinReorderTest.java.
union node's result exprs should be substitued by child node's smap first, then the following "computePassthrough" method would have correct information to do its job.
* [enhancement](ldap) optimize LDAP authentication.
1. Support caching LDAP user information.
2. HTTP authentication supports LDAP.
3. LDAP temporary users support default user property.
4. LDAP configuration supports the `admin show config` and `admin set config` commands.
