- enable CBO stage in Nereids
- use the `chooseBestPlan()` to get the best plan
- add a new rule JoinCommuteProject
- test the stage by JoinCommute rule
When Sum's child is Decimal, Return Double Type by mistake lead to result error, so we should keep the return type to decimal when the child expression's type is decimal.
this pr do 2 refactor
1. remove useless parameter from `Plan#computeOutput`
2. refactor memo.copyIn
It the past, `memo.copyIn` has complex logic to process init, rewrite and copyIn, It's difficult to understand and easy to meet bug and leak memory for some unreachable group/groupExpression.
So I separate it into three methods:
1. `Memo.init` for init Memo by LogicalPlan
2. `Memo.doRewrite` for rewrite
3. `Memo.doCopyIn` for exploration and implementation
And separate the UT into 3 files
1. `MemoInitTest`
2. `MemoRewriteTest`
3. `MemoCopyInTest`
I have added a lots of UT for `Memo.rewrite`, and add some unreachable DAG check in the PlanChecker, when the plan is changed.
We have added logical project before, but to actually finish the prune to reduce the data IO, we need to add related supports in translator and BE.
This PR:
- add projections on each ExecNode in BE
- translate PhysicalProject into projections on PlanNode in FE
- do column prune on ScanNode in FE
Co-authored-by: HappenLee <happenlee@hotmail.com>
We can skip aggregate on replace column, otherwise it would generate
wrong result. e.g. a row in UNIQUE is deleted by delte_sign_column,
then it would be returned.
Copy most of profiles from VOlapScanNode and VOlapScanner to NewOlapScanNode and NewOlapScanner.
Fix some blocking bug of new scan framework.
TODO:
Memtracker
Opentelemetry spen
The new framework is still disabled by default, so it will not effect other feature.
Array column should not be distributed key or group by or aggregate key, we should forbid it before create table.
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
Add one expression rewrite rule:
rewrite InPredicate to an EqualTo Expression, if there exists exactly one element in InPredicate Options.
Examples:
1. where A in (x) ==> where A = x
2. where A not in (x) ==> where not A = x
In BE, There is an implicit convention that HashJoinNode's left child's output Slot must before right child's output slot in intermediateTuple.
However, after we do commute rule on join plan in Nereids, this convention will be broken and cause core dump in BE.
There are two way to fix this problem:
1. add a project on join after we do commute
2. reorder output of join node when we do translate
Since we cannot translate project yet because BE projection support is on going(#11842). So we use second way to fix it now. After the project translation could work correctly, we should use the first way to fix it.
In current implementation, we detect invalid slot at execute phase. At execute phase, it is hard to get useful information for further debug. This pr moves error detection ahead to prepare phase, so that we can log related tuple descriptors.
Refactor TaggableLogger
Refactor status handling in agent task:
Unify log format in TaskWorkerPool
Pass Status to the top caller, and replace some OLAPInternalError with more detailed error message Status
Premature return with the opposite condition to reduce indention
Problem:
1. `enable_array_type` is masterOnly;
2. dynamic open config only affect FE MASTER
`admin set frontend config("enable_array_type"="true");`
3. query in FE FOLLOWER will fail, because of `enable_array_type` is false in FE FOLLOWER
`select * from table_with_array `
Solution:
Only check `enable_array_type` while creating new tables with array column.
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
1. Rewrite Filter(Project) to Project(Filter) to make sure when do partition prune the tree looks like this: Project(Filter(OlapScan)).
2. Enable the MergeConsecutiveProject MergeConsecutiveFilter rules.
3. prune range partition just like what Legacy Planner do.
