Instead of add a cast function on literal, we directly change the literal type. This change could save cast execution time and memory.
For example:
In SQL:
"CASE WHEN l_orderkey > 0 THEN ...", 0 is a TinyIntLiteral.
Before this PR:
"CASE WHEN l_orderkey > CAST (TinyIntLiteral(0) AS INT)`
With this PR:
"CASE WHEN l_orderkey > IntegerLiteral(0)"
We used output list to compare two LogicalProperties before. Since join reorder will change the children order of a join plan and caused output list changed. the two join plan will not equals anymore in memo although they should be. So we must add a project on the new join to keep the LogicalProperties the same.
This PR changes the equals and hashCode funtions of LogicalProperties. use a set of output to compare two LogicalProperties. Then we do not need add the top peoject anymore. This help us keep memo simple and efficient.
Template for building internal query SQL statements，it mainly used for statistics module. After the template is defined, the executable statement will be built after the given parameters.
For example, template and parameters:
- template: `SELECT ${col} FROM ${table} WHERE id = ${id};`,
- parameters: `{col=colName, table=tableName, id=1}`
- result sql: `SELECT colName FROM tableName WHERE id = 1;`
usage:
```
String template = "SELECT * FROM ${table} WHERE id = ${id};";
Map<String, String> params = new HashMap<>();
params.put("table", "table0");
params.put("id", "123");
// result: SELECT * FROM table0 WHERE id = 123;
String result = InternalSqlTemplate.processTemplate(template, params);
```
There is a problem with StatisticsTaskScheduler. The peek() method obtains a reference to the same task object, but the for-loop executes multiple removes.
Execute SQL query statements internally(in FE). Internal-query mainly used for statistics module, FE obtains statistics by SQL from BE, such as column maximum value, minimum value, etc.
This is a tool module as statistics, it will not affect the original code, also will not affect the use of users.
The simple usage process is as follows(the following code does no exception handling):
```
String dbName = "test";
String sql = "SELECT * FROM table0";
InternalQuery query = new InternalQuery(dbName, sql);
InternalQueryResult result = query.query();
List<ResultRow> resultRows = result.getResultRows();
for (ResultRow resultRow : resultRows) {
List<String> columns = resultRow.getColumns();
for (int i = 0; i < resultRow.getColumns().size(); i++) {
resultRow.getColumnIndex(columns.get(i));
resultRow.getColumnName(i);
resultRow.getColumnType(columns.get(i));
resultRow.getColumnType(i);
resultRow.getColumnValue(columns.get(i));
resultRow.getColumnValue(i);
}
}
```
Refactor the scanners for hms external catalog, work in progress.
Use VFileScanner, will remove NewFileParquetScanner, NewFileOrcScanner and NewFileTextScanner after fully tested.
Query for parquet file has been tested, still need to add readers for orc file, text file and load logic as well.
1. Add all slots used by onClause in project
```
(A & B) & C like
join(hash conjuncts: C.t2 = A.t2)
|---project(A.t2)
| +---join(hash conjuncts: A.t1 = B.t1)
| +---A
| +---B
+---C
transform to (A & C) & B
join(hash conjuncts: A.t1 = B.t1)
|---project(A.t2)
| +---join(hash conjuncts: C.t2 = A.t2)
| +---A
| +---C
+---B
```
But projection just include `A.t2`, can't find `A.t1`, we should add slots used by onClause when projection exist.
2. fix join reorder mark
Add mark `LAsscom` when apply `LAsscom`
3. remove slotReference
use `Slot` instead of `SlotReference` to avoid cast.
This PR fix:
2 Backends.
Create tables with colocation group, 1 replica.
Decommission one of Backends.
The tablet on decommissioned Backend is not reduced.
This is a bug of ColocateTableCheckerAndBalancer.
Fix _delete_sign_idx and _seq_col_idx when append_column or build_schema when load.
Tablet schema cache support recycle when schema sptr use count equals 1.
Add a http interface for flink-connector to sync ddl.
Improve tablet->tablet_schema() by max_version_schema.
This pr did these things:
1. Change the nullable mode of 'from_unixtime' and 'parse_url' from DEPEND_ON_ARGUMENT to ALWAYS_NULLABLE, which nullable configuration was missing previously.
2. Add some new interfaces for origin NullableMode. This change inspired by the grammar of scala's mix-in trait, It help us to quickly understand the traits of function without read the lengthy procedural code and save the work to write some template code, like `class Substring extends ScalarFunction implements ImplicitCastInputTypes, PropagateNullable`. These are the interfaces:
- PropagateNullable: equals to NullableMode.DEPEND_ON_ARGUMENT
- AlwaysNullable: equals to NullableMode.ALWAYS_NULLABLE
- AlwaysNotNullable: equals to NullableMode.ALWAYS_NOT_NULLABLE
- others ComputeNullable: equals to NullableMode.CUSTOM
3. Add `GenerateScalarFunction` to generate nereids-style function code from legacy functions, but not actual generate any new function class yet, because the function's trait is not ready for use. I need add some traits for the legacy function's CompareMode and NonDeterministic, this thought is the same as ComputeNullable.
For eliminate all unessential cross join in TPC-H benchmark, this PR:
1. push all predicates that can be push down through join before do ReorderJoin rule.
Then we could eliminate all cross join that can be eliminated in ReorderJoin rule since this rule need matching a LogicalFilter as a root pattern. (Q2, Q15, Q16, Q17, Q18)
2. enable expression optimization rule - extract common expression. (Q19)
3. fix cast translate failed. (Q19)
test sql: TPC-H q21
```
select count(*)
from lineitem l3 right anti join lineitem l1
on l3.l_orderkey = l1.l_orderkey and l3.l_suppkey <> l1.l_suppkey;
```
if we have other join conjuncts, we have to put all slots from left and right into `slotReferenceMap` instead of `hashjoin.getOutput()`
After splitting intermediate tuple and output tuple, we meet several issues in regression test. And hence, we make following changes:
1. since translating project will replace underlying hash-join node's output tuple, we add PhysicalHashJoin.shouldTranslateOutput
2. because PhysicalPlanTranslator will merge filter and hashJoin, we add PhysicalHashJoin.filterConjuncts and translate filter conjuncts in physicalHashJoin
3. In this pr, we set HashJoinNode.hashOutputSlotIds properly when using nereids planner.
4. in order to be compatible with BE, in substring function, nullable() returns true
When we do type coercion on CaseWhen expression, such as sql like this:
```
CASE WHEN n_nationkey > 1 THEN n_regionkey ELSE 0 END
```
The ELSE part 0 need do type coercion as CAST (0 AS INT). But we miss it in PR #11802
This PR add runtime filter to Nereids planner. Now only support push through join node and scan node.
TODO:
1. current support inner join, cross join, right outer join, and will support other join type in future.
2. translate left outer join to inner join if there are inner join ancestors.
3. some complex situation cannot be handled now, see more details in test case: testPushDownThroughJoin.
4. support src key is aggregate group key.
NamedExpressionUtil::clear should reset the nextId rather than create a new IdGenerator<ExprId> because the old one may be referenced by other objects and it may cause some cases start in a dirty environment when we run test cases in package.
support distinct count with group by clause.
for example:
SELECT count(distinct c_custkey + 1) FROM customer group by c_nation;
TODO: support distinct count without group by clause.
This PR changed some interfaces to avoid unsafe cast.
- Modify `Plan.getExpressions()`'s return type from `List<Expression>` to `List<? extends Expression>`
Return projects (type is a list of named expression) in `getExpressions` can avoid unsafe cast. See `LogicalProject.getExpression()` as an example.
- Modify `EmptyRelation.getProjects()`'s return type from `List<NamedExpression>` to `List<? extends NamedExpression>`
Creating empty relation with a list of slots can avoid unsafe cast. See the `EliminateLimit` rule for example.
In Nereids, we could not distinguish two relation from same table in one PlanTree.
This lead to some trick code to process them when do plan. Such as a separate branch to do equals in GroupExpression.
This PR add RelationId to LogicalRelation and PhysicalRelation. Then all relations equals function will compare RelationId to help us distinguish two relation from same table.
TODO:
add relation id to UnboundRelation, UnboundOneRowRelation, LogicalOneRowRelation, PhysicalOneRowRelation.
A channel is closed when a timeout or exception happens, if only
one stub is used, then all query would fail.
If we dont close the channel, sometimes grpc-java stuck without sending
any rpc.
1. For some related rules, we need to execute them together to get the expected plan.
2. Add session variables to avoid fallback to stale planner when running regression tests of nereids for piggyback.
