Define a new file scanner node for hms table in be.
This file scanner node is different from broker scan node as blow:
1. Broker scan node will define src slot and dest slot, there is two memory copy in it: first is from file to src slot
and second from src to dest slot. Otherwise FileScanNode only have one stemp memory copy just from file to dest slot.
2. Broker scan node will read all the filed in the file to src slot and FileScanNode only read the need filed.
3. Broker scan node will convert type into string type for src slot and then use cast to convert to dest slot type,
but FileScanNode will have the final type.
Now FileScanNode is a standalone code, but we will uniform the file scan and broker scan in the feature.
In order to complete the ssb test, temporarily increase the implementation of costAndEnforcerJob, and create an OptimizeGroupjob for all children of the group.
Add more implmentation rules.
Current some `logical` and `physical` operator is different. I change some code to make them match.
Implementation
- Sort：only heap sort
- Agg
- OlapScan
doris on es8 can not work, because type change. The use of type is no longer recommended in es7,
and support for type has been removed from es8.
1. /_mapping not support include_type_name
2. /_search not support use type
To integrate the nereids optimizer with new SQLParser and Planner to existing SQL process framework, I abstract a interface which named "Planner" and let the Both planner from nereids and stale optimizer implement it, to disguish it
with origin Planner, I rename the Planner to OriginalPlanner.
As we don't want to impact the existing logic too much, I defined a LogicalPlanAdapter to adapt the logicalPlan that is the output of the new paser to the existing code.
Besides, as the MySQL protocol supports sending multiple statements in one packet, so I add Nereids#SparseSQL method to handle this properly.
In a vectorized scenario, the query plan will generate a new tuple for the join node.
This tuple mainly describes the output schema of the join node.
Adding this tuple mainly solves the problem that the input schema of the join node is different from the output schema.
For example:
1. The case where the null side column caused by outer join is converted to nullable.
2. The projection of the outer tuple.
We do not generate real java doc. All java doc comments is used to help to understand the code logic more easily.
So we need loose java doc style check. Remove period character check in summary java doc check rule.
A temp DataSourceMgr is created when catalog intializes, and will be
updated in the loadCluster process. However, the loadDatasource process
will create a new DataSourceMgr with no cluster updated, and write such
DatasourceMgr to image. Finally, when the doris starts to recover, no
cluster is available.
Only do implementation in cascades.
In order to achieve the ssb test, at this stage, temporarily comment out the explore rule process in cascades.
It needs to be added in the future.
PropertyAnalyzer.analyzeDataProperty(Map<String, String> properties, final DataProperty oldDataProperty) has something not suitable.
Parameter oldDataProperty is the old DataProperty, properties should be used to replace some of its members.
If properties has no some members, old ones need to be left, but not be set to default value.
Function modifyPartitionsProperty() uses analyzeDataProperty(), but create a new DataProperty again, it is duplicate.
Add related expressions and AggPlan and SortPlan.
This PR includes the addition of the following expression types:
Arithmetic/BetweenPredicate/CompoundPredicate/FunctionCall.
PlanNode:
LogicalAggregation/LogicalSort
Add a ut to verify related expressions and node parsing
Supported:
1. Change FeMetaVersion to 111, compatible with upgrade from 110.
2. Add catalog level privileges, and degrade global level privileges to catalog level if FeMetaVersion < 111.
3. Support 'show all grants', 'show roles' statement.
4. Previous version of SQL syntax.
Todo:
1. three-segment format catalog.database.table in SQL syntax.
2. User document for the unified authority management of datalake.
3. LDAP services to provide authentication.
As the statistics framework could not be reused by new optmizer before, so I abstract some interface to make it reusable.
1. Make Slot extends the Id
2. Add new interface:ExprStats,PlanStats
3. Move definition of PlanNode.NodeType to statistics sub-directory
Add remote storage policy config for create table properties. It will set storage policy for table and partitions in `CREATE TABLE` and `ALTER TABLE`.
This policy will be used when partition is being migrated from local to remote.
grammy:
1.
`CREATE TABLE TblPxy1
(...)
ENGINE=olap
DISTRIBUTED BY HASH (aa) BUCKETS 1
PROPERTIES(
"remote_storage_policy" = "testPolicy3"
);`
2.
`ALTER TABLE TblPxy01 SET ("remote_storage_policy" = "testPolicy3");`
3.
`ALTER TABLE TblPxy01 MODIFY PARTITION p2 SET ("remote_storage_policy" = "testPolicy3");`
FEFunctionSignature do not support ArrayType as args, then following SQL failed:
`> select array_contains([1,2,3], 1);`
ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: org.apache.doris.catalog.ArrayType cannot be cast to org.apache.doris.catalog.ScalarType
Currently, when setting variables with `global` keywords, it will not affect the
current session variable's value. That is always make user confused.
This CL mainly changes:
1. Change session variable when set global variable
