1. Convert child expressions in InPredicate to column type and discard child expressions in them that cannot be converted exactly.
2. Fix the bug of ColumnRange exception caused by InPredicate child expressions type conversion.
3. Fix the problem that the tablet could not be hit due caused by InPredicate child expressions type conversion.
Issue Number: close#9627 , #9628
This PR introduce two essentials for Nereids
1. pattern match iterator used in memo
pattern match iterator is implemented by two iterators nested within each other: GroupExpressionIterator and GroupIterator.
GroupExpressionIterator use GroupIterator to get all children Plan which matching pattern and use them as children to generate pattern matched plan.
GroupIterator use GroupExpressionIterator to get all pattern matched Plan related to GroupExpressions in itself.
2. plan rewrite framework for memo
Rewrite framework is implemented by two jobs: RewriteTopDownJob and RewriteBottomUpJob
Both of them takes a group, a set of rules that need to be applied, and a context as construction parameters.
RewriteTopDownJob apply these jobs from top to down one by one.
RewriteBottomUpJob apply these jobs from bottom to up one by one.
When one rule rewrites plan tree at a plan node. This plan node will be applied all rules again until no rules can rewrite it.
This CL mainly changes:
1. Broker Load
When assigning backends, use user level resource tag to find available backends.
If user level resource tag is not set, broker load task can be assigned to any BE node,
otherwise, task can only be assigned to BE node which match the user level tags.
2. Routine Load
The current routine load job does not have user info, so it can not get user level tag when assigning tasks.
So there are 2 ways:
1. For old routine load job, use tags of replica allocation info to select BE nodes.
2. For new routine load job, the user info will be added and persisted in routine load job.
Close#9623
Summary:
This pr refactor plan node into plan + operator.
In the previous version in nereids, a plan node consists of children and relational algebra, e.g.
```java
class LogicalJoin extends LogicalBinary {
private Plan left, right;
}
```
This structure above is easy to understand, but it difficult to optimize `Memo.copyIn`: rule generate complete sub-plan,
and Memo must compare the complete sub-plan to distinct GroupExpression and hurt performance.
First, we need change the rule to generate partial sub-plan, and replace some children plan to a placeholder, e.g. LeafOp in Columbia optimizer. And then mark some children in sub-plan to unchanged, and bind the relate group, so don't have to compare and copy some sub-plan if relate group exists.
Second, we need separate the origin `Plan` into `Plan` and `Operator`, which Plan contains children and Operator, and Operator just denote relation relational algebra(no children/ input field). This design make operator and children not affect each other. So plan-group binder can generate placeholder plan(contains relate group) for the sub-query, don't have to generate current plan node case by case because the plan is immutable(means generate a new plan with replace children). And rule implementer can reuse the placeholder to generate partial sub-plan.
Operator and Plan have the similar inheritance structure like below. XxxPlan contains XxxOperator, e.g. LogicalBinary contains a LogicalBinaryOperator.
```
TreeNode
│
│
┌───────┴────────┐ Operator
│ │ │
│ │ │
│ │ │
▼ ▼ ▼
Expression Plan PlanOperator
│ │
│ │
┌───────────┴─────────┐ │
│ │ ┌───────────┴──────────────────┐
│ │ │ │
│ │ │ │
▼ ▼ ▼ ▼
LogicalPlan PhysicalPlan LogicalPlanOperator PhysicalPlanOperator
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
├───►LogicalLeaf ├──►PhysicalLeaf ├──► LogicalLeafOperator ├───►PhysicalLeafOperator
│ │ │ │
│ │ │ │
│ │ │ │
├───►LogicalUnary ├──►PhysicalUnary ├──► LogicalUnaryOperator ├───►PhysicalUnaryOperator
│ │ │ │
│ │ │ │
│ │ │ │
└───►LogicalBinary └──►PhysicalBinary └──► LogicalBinaryOperator └───►PhysicalBinaryOperator
```
The concrete operator extends the XxxNaryOperator, e.g.
```java
class LogicalJoin extends LogicalBinaryOperator;
class PhysicalProject extends PhysicalUnaryOperator;
class LogicalRelation extends LogicalLeafOperator;
```
So the first example change to this:
```java
class LogicalBinary extends AbstractLogicalPlan implements BinaryPlan {
private Plan left, right;
private LogicalBinaryOperator operator;
}
class LogicalJoin extends LogicalBinaryOperator {}
```
Under such changes, Rule must build the plan and operator as needed, not only the plan like before.
for example: JoinCommutative Rule
```java
public Rule<Plan> build() {
// the plan override function can automatic build plan, according to the Operator's type,
// so return a LogicalBinary(LogicalJoin, Plan, Plan)
return innerLogicalJoin().then(join -> plan(
// operator
new LogicalJoin(join.op.getJoinType().swap(), join.op.getOnClause()),
// children
join.right(),
join.left()
)).toRule(RuleType.LOGICAL_JOIN_COMMUTATIVE);
}
```
Due to the current architecture, predicate derivation at rewrite cannot satisfy all cases,
because rewrite is performed on first and then where, and when there are subqueries, all cases cannot be derived.
So keep the predicate pushdown method here.
eg.
select * from t1 left join t2 on t1 = t2 where t1 = 1;
InferFiltersRule can't infer t2 = 1, because this is out of specification.
The expression(t2 = 1) can actually be deduced to push it down to the scan node.
* [Refactor][Bug-Fix][Load Vec] Refactor code of basescanner and vjson/vparquet/vbroker scanner
1. fix bug of vjson scanner not support `range_from_file_path`
2. fix bug of vjson/vbrocker scanner core dump by src/dest slot nullable is different
3. fix bug of vparquest filter_block reference of column in not 1
4. refactor code to simple all the code
It only changed vectorized load, not original row based load.
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
Hive and trino/presto would automatically trim the trailing spaces but Doris doesn't.
This would cause different query result with hive.
Add a new session variable "trim_tailing_spaces_for_external_table_query".
If set to true, when reading csv from broker scan node, it will trim the tailing space of the column
unnecessary cast will be added on children in CaseExpr because use symbolized equal to compare to `Expr`'s type.
it will lead to expression compare mistake and then lead to expression substitute failed when use `ExprSubstitutionMap`
Push down predicate past aggregate cannot push down predicate past 2 phase aggregate.
origin plan is like this:
```
second phase agg (conjuncts on olap scan node tuples)
|
first phase agg
|
olap scan node
```
should be optimized to
```
second phase agg
|
first phase agg
|
olap scan node (conjuncts on olap scan node tuples)
```
1. Remove session variable "enable_lateral_view"
2. Remove Fe config: enable_materialized_view
3. Remove Fe config: enable_create_sync_job
4. Fe config dynamic_partition_enable is only used for disable dynamic partition scheduler.
Fixed#9529
When the lateral view based on a inline view which belongs to a view,
Doris could not resolve the column of lateral view in query.
When a query uses a view, it mainly refers to the string representation of the view.
That is, if the view's string representation is wrong, the view is wrong.
The string representation of the inline view lacks the handling of the lateral view.
This leads to query errors when using such views.
This PR mainly fixes the string representation of inline views.
Issue Number: close#9403
set below rules' severity to error and format code according check info.
a. Merge conflicts unresolved
b. Avoid using corresponding octal or Unicode escape
c. Avoid Escaped Unicode Characters
d. No Line Wrap
e. Package Name
f. Type Name
g. Annotation Location
h. Interface Type Parameter
i. CatchParameterName
j. Pattern Variable Name
k. Record Component Name
l. Record Type Parameter Name
m. Method Type Parameter Name
n. Redundant Import
o. Custom Import Order
p. Unused Imports
q. Avoid Star Import
r. tab character in file
s. Newline At End Of File
t. Trailing whitespace found
binlog load Because txn exceeds the default value, resume is a failure,
and a friendly prompt message is given to the user, instead of prompting success now,
it still fails after a while, and the user will feel inexplicable
Issue Number: close#9468
