9db56201a6
This pr refactor the rewrite framework from memo to plan tree, and speed up the analyze/rewrite stage. Changes: - abandoned memo in the analysis/rewrite stage, so that we can skip some actions, like new GroupExpression, distinct GroupExpression in the memo(high cost), update children to GroupPlan - change the most of rules to static rule, so that we can skip initialize lots of rules in Analyzer/Rewriter at every query. but some rules need context, like visitor rule, create rule at the runtime make it is easy to use, so make `custom` rule can help us to create it. - remove the `logger` field in the Job, Job are generated in large quantities at runtime, we don't need to use logger so save huge time to initialize logger. - skip some rule as far as possible, e.g. `SelectMaterializedIndexWithoutAggregate`, skip select mv if the table not exist rullup. - add some caches for frequent operation, like get Job.getDisableRules, Plan.getUnboundExpression - new bottom up rewrite rule, it can keep traverse multiple new plan which return by rules, this feature depends on `Plan.mutableState`, it is necessary to add this variable field for plan. if the plan is fully immutable, we must use withXxx to renew the plan and set the state for it, this take more runtime overhead and developing workload. another reason is we need multiple mutable state, e.g. whether is applied the rule, whether this plan is manage by the rewrite framework. the good side of mutable state is efficient, but I suggest we don't direct use mutable state in the rule as far as possible, if we need use it, please wrap the mutable state in the framework to update and release it correctly. a good example is `AppliedAwareRuleCondition`, it can update and get the state: whether this plan is applied to a rule before. - merge some rules, invoke multiple rules in one traverse - refactor the `EliminateUnnecessaryProject` by CustomRewritor, fix the problem which eliminate some Project which decided the query output order, the case is limit(project), sort(project). TODO: add trace for new rewrite framework benchmark: legacy optimizer: ``` +-----------+---------------+---------------+---------------+ | SQL ID | avg | min | max | +-----------+---------------+---------------+---------------+ | SQL 1 | 1.39 ms | 0 ms | 9 ms | | SQL 2 | 1.38 ms | 0 ms | 10 ms | | SQL 3 | 2.05 ms | 1 ms | 18 ms | | SQL 4 | 0.89 ms | 0 ms | 9 ms | | SQL 5 | 1.74 ms | 1 ms | 11 ms | | SQL 6 | 2.00 ms | 1 ms | 13 ms | | SQL 7 | 1.83 ms | 1 ms | 15 ms | | SQL 8 | 0.92 ms | 0 ms | 7 ms | | SQL 9 | 2.60 ms | 1 ms | 19 ms | | SQL 10 | 3.54 ms | 2 ms | 28 ms | | SQL 11 | 3.04 ms | 1 ms | 18 ms | | SQL 12 | 3.26 ms | 2 ms | 16 ms | | SQL 13 | 1.10 ms | 0 ms | 10 ms | | SQL 14 | 2.90 ms | 1 ms | 13 ms | | SQL 15 | 1.18 ms | 0 ms | 9 ms | | SQL 16 | 1.05 ms | 0 ms | 13 ms | | SQL 17 | 1.03 ms | 0 ms | 7 ms | | SQL 18 | 0.94 ms | 0 ms | 7 ms | | SQL 19 | 1.47 ms | 0 ms | 13 ms | | SQL 20 | 0.47 ms | 0 ms | 4 ms | | SQL 21 | 0.54 ms | 0 ms | 5 ms | | SQL 22 | 3.34 ms | 1 ms | 19 ms | | SQL 23 | 7.97 ms | 4 ms | 44 ms | | SQL 24 | 11.11 ms | 7 ms | 28 ms | | SQL 25 | 0.98 ms | 0 ms | 8 ms | | SQL 26 | 0.83 ms | 0 ms | 7 ms | | SQL 27 | 0.93 ms | 0 ms | 16 ms | | SQL 28 | 2.19 ms | 1 ms | 18 ms | | SQL 29 | 3.23 ms | 1 ms | 20 ms | | SQL 30 | 59.99 ms | 51 ms | 81 ms | | SQL 31 | 2.65 ms | 1 ms | 18 ms | | SQL 32 | 2.47 ms | 1 ms | 17 ms | | SQL 33 | 2.30 ms | 1 ms | 16 ms | | SQL 34 | 0.66 ms | 0 ms | 8 ms | | SQL 35 | 0.63 ms | 0 ms | 6 ms | | SQL 36 | 2.25 ms | 1 ms | 15 ms | | SQL 37 | 5.97 ms | 3 ms | 20 ms | | SQL 38 | 5.73 ms | 3 ms | 21 ms | | SQL 39 | 6.32 ms | 4 ms | 23 ms | | SQL 40 | 8.61 ms | 5 ms | 35 ms | | SQL 41 | 6.29 ms | 4 ms | 28 ms | | SQL 42 | 6.04 ms | 4 ms | 15 ms | | SQL 43 | 5.81 ms | 3 ms | 16 ms | +-----------+---------------+---------------+---------------+ | TOTAL AVG | 4.22 ms | 2.47 ms | 17.05 ms | | TOTAL SUM | 181.62 ms | 106 ms | 733 ms | +-----------+---------------+---------------+---------------+ ``` nereids with memo rewrite framework(old): ``` +-----------+---------------+---------------+---------------+ | SQL ID | avg | min | max | +-----------+---------------+---------------+---------------+ | SQL 1 | 3.61 ms | 1 ms | 20 ms | | SQL 2 | 3.47 ms | 2 ms | 16 ms | | SQL 3 | 3.27 ms | 1 ms | 18 ms | | SQL 4 | 2.23 ms | 1 ms | 12 ms | | SQL 5 | 3.60 ms | 1 ms | 20 ms | | SQL 6 | 2.73 ms | 1 ms | 17 ms | | SQL 7 | 3.04 ms | 1 ms | 23 ms | | SQL 8 | 3.53 ms | 2 ms | 20 ms | | SQL 9 | 3.74 ms | 2 ms | 22 ms | | SQL 10 | 3.66 ms | 2 ms | 18 ms | | SQL 11 | 3.93 ms | 2 ms | 15 ms | | SQL 12 | 4.85 ms | 2 ms | 27 ms | | SQL 13 | 4.41 ms | 2 ms | 28 ms | | SQL 14 | 5.16 ms | 2 ms | 41 ms | | SQL 15 | 4.33 ms | 2 ms | 33 ms | | SQL 16 | 4.94 ms | 2 ms | 51 ms | | SQL 17 | 3.27 ms | 1 ms | 25 ms | | SQL 18 | 2.78 ms | 1 ms | 22 ms | | SQL 19 | 3.51 ms | 1 ms | 42 ms | | SQL 20 | 1.84 ms | 1 ms | 13 ms | | SQL 21 | 3.47 ms | 1 ms | 66 ms | | SQL 22 | 5.21 ms | 2 ms | 29 ms | | SQL 23 | 5.55 ms | 3 ms | 25 ms | | SQL 24 | 4.21 ms | 2 ms | 28 ms | | SQL 25 | 3.47 ms | 1 ms | 23 ms | | SQL 26 | 3.03 ms | 2 ms | 21 ms | | SQL 27 | 3.07 ms | 1 ms | 17 ms | | SQL 28 | 4.51 ms | 3 ms | 22 ms | | SQL 29 | 4.97 ms | 3 ms | 21 ms | | SQL 30 | 11.95 ms | 8 ms | 33 ms | | SQL 31 | 3.92 ms | 2 ms | 23 ms | | SQL 32 | 3.74 ms | 2 ms | 15 ms | | SQL 33 | 3.62 ms | 2 ms | 22 ms | | SQL 34 | 4.60 ms | 1 ms | 55 ms | | SQL 35 | 3.47 ms | 2 ms | 25 ms | | SQL 36 | 3.34 ms | 2 ms | 18 ms | | SQL 37 | 4.77 ms | 2 ms | 23 ms | | SQL 38 | 4.44 ms | 2 ms | 39 ms | | SQL 39 | 4.52 ms | 2 ms | 23 ms | | SQL 40 | 5.50 ms | 3 ms | 30 ms | | SQL 41 | 5.01 ms | 2 ms | 24 ms | | SQL 42 | 4.32 ms | 2 ms | 24 ms | | SQL 43 | 4.29 ms | 2 ms | 42 ms | +-----------+---------------+---------------+---------------+ | TOTAL AVG | 4.11 ms | 1.91 ms | 26.30 ms | | TOTAL SUM | 176.88 ms | 82 ms | 1131 ms | +-----------+---------------+---------------+---------------+ ``` nereids with plan tree rewrite framework(new): ``` +-----------+---------------+---------------+---------------+ | SQL ID | avg | min | max | +-----------+---------------+---------------+---------------+ | SQL 1 | 3.21 ms | 1 ms | 18 ms | | SQL 2 | 3.99 ms | 1 ms | 76 ms | | SQL 3 | 2.93 ms | 1 ms | 21 ms | | SQL 4 | 2.13 ms | 1 ms | 21 ms | | SQL 5 | 2.43 ms | 1 ms | 30 ms | | SQL 6 | 2.08 ms | 1 ms | 11 ms | | SQL 7 | 2.03 ms | 1 ms | 11 ms | | SQL 8 | 2.27 ms | 1 ms | 22 ms | | SQL 9 | 2.42 ms | 1 ms | 16 ms | | SQL 10 | 2.65 ms | 1 ms | 14 ms | | SQL 11 | 2.78 ms | 1 ms | 14 ms | | SQL 12 | 3.09 ms | 1 ms | 19 ms | | SQL 13 | 2.33 ms | 1 ms | 13 ms | | SQL 14 | 2.66 ms | 1 ms | 16 ms | | SQL 15 | 2.34 ms | 1 ms | 15 ms | | SQL 16 | 2.04 ms | 1 ms | 30 ms | | SQL 17 | 2.09 ms | 1 ms | 17 ms | | SQL 18 | 1.87 ms | 1 ms | 15 ms | | SQL 19 | 2.21 ms | 1 ms | 50 ms | | SQL 20 | 1.32 ms | 0 ms | 12 ms | | SQL 21 | 1.63 ms | 1 ms | 11 ms | | SQL 22 | 2.75 ms | 1 ms | 30 ms | | SQL 23 | 3.44 ms | 2 ms | 17 ms | | SQL 24 | 2.01 ms | 1 ms | 14 ms | | SQL 25 | 1.58 ms | 1 ms | 11 ms | | SQL 26 | 1.53 ms | 0 ms | 13 ms | | SQL 27 | 1.62 ms | 1 ms | 12 ms | | SQL 28 | 2.90 ms | 1 ms | 21 ms | | SQL 29 | 3.04 ms | 2 ms | 17 ms | | SQL 30 | 10.54 ms | 7 ms | 49 ms | | SQL 31 | 2.61 ms | 1 ms | 21 ms | | SQL 32 | 2.42 ms | 1 ms | 14 ms | | SQL 33 | 2.13 ms | 1 ms | 14 ms | | SQL 34 | 1.69 ms | 1 ms | 14 ms | | SQL 35 | 1.87 ms | 1 ms | 15 ms | | SQL 36 | 2.37 ms | 1 ms | 21 ms | | SQL 37 | 3.06 ms | 1 ms | 15 ms | | SQL 38 | 4.09 ms | 1 ms | 31 ms | | SQL 39 | 5.81 ms | 2 ms | 43 ms | | SQL 40 | 4.55 ms | 2 ms | 34 ms | | SQL 41 | 3.49 ms | 1 ms | 20 ms | | SQL 42 | 2.75 ms | 1 ms | 26 ms | | SQL 43 | 2.81 ms | 1 ms | 14 ms | +-----------+---------------+---------------+---------------+ | TOTAL AVG | 2.78 ms | 1.19 ms | 21.35 ms | | TOTAL SUM | 119.56 ms | 51 ms | 918 ms | +-----------+---------------+---------------+---------------+ ```
# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # fe-common This module is used to store some common classes of other modules. # spark-dpp This module is Spark DPP program, used for Spark Load function. Depends: fe-common # fe-core This module is the main process module of FE. Depends: fe-common, spark-dpp