Problem and Cause:
In original planner, date_add function would choose different data type of datetime or datev2. Which when original planner choose datev2 as constant date type. Fe could not folding date_add function because missing of function signature of folding constant date_add(datev2, int)
Solved:
Add corresponding function signatures of date_add/months_add/years_add in original planner
baseTables in MTMVRelation stores all baseTables in the nested materialized view,
now adding baseTablesOneLevel to only store the baseTables at the current level.
* [refactor](Nereids)refactor runtime filter generator (#34275)
1. unify the process of generating rf for hash join and for nested loop join
2. fix some bugs in generating rf
3. remove some duplicated check
(cherry picked from commit 07267faac0d9c6ef3bb1fd4ee101b4c761c8a2f2)
* [refactor](nereids) do not deny a runtime filter by removing an entry in aliasMap (#34559)
in current version, there are 2 approaches to verify whether a join condition can be used to generate a runtime filter, they are
1. remove the output slot from aliasMap
2. pushDownVisitor.visit(...) return false
the 1st approach has some drawbacks, we prefer to the 2ed approach.
In this pr, all the cases are handled by the 2ed approach, and remove the related code for the 1st approach.
(cherry picked from commit a29082bf31e66efa2df193b38347e610f2bf7464)
* rebase
we have an order reserved mappping from string to double.
for string column A, we have double values for A.min and A.max.
when estimating A<"abc", A.min/max could be used to judge whether 'abc' is between A.min and A.max, but it cannot be used to do range estimation. suppose "abc" is mapped to double x. if we compute selectivity by formula "sel = (x-A.min)/(A.max-A.min)", we are likely to obtain extreme values.
1. let AggStateType extends Type
2. remove useless interface isFixedLengthType and supportsTablePartitioning
3. let MapType implement interface isSupported
4. let VariantType extends ScalarType
closeOrReleaseResources(), which is responsible for safely closing or releasing the stmtExecutor, command, and ctx objects. The method follows a null check pattern to avoid NullPointerExceptions and ensures that resources are properly cleaned up when they are no longer needed. This improves code readability and maintains a consistent approach to resource management.
In NormalizeRepeat, three parts of the outputExpression of LogicalRepeat need to be pushed down and outputted by bottom project: flattenGroupingSetExpr, argumentsOfGroupingScalarFunction, argumentsOfAggregateFunction.
In the original code, use these three parts to rewrite the outputExpressions of LogicalRepeat to slots.This can cause problems in some cases, for example:
```sql
SELECT
ROUND( SUM(pk + 1) - 3) col_alias1,
pk + 1 AS col_alias3
FROM
table_20_undef_partitions2_keys3_properties4_distributed_by53
GROUP BY
GROUPING SETS ((pk), ()) ;
```
The three parts expression needed to be pushed down are: pk, pk+1. The original code use pk+1 to rewrite the pk + 1 AS col_alias3 to slot. But the pk+1 is not in the list of grouping outputs, and then report error.
This pr change the rewrite process, divide the expression needed to be pushed down into 2 parts: one is (flattenGroupingSetExpr) and the other one is (argumentsOfGroupingScalarFunction, argumentsOfAggregateFunction).
and use the flattenGroupingSetExpr rewrite all LogicalRepeat outputExpressions, and use the argumentsOfGroupingScalarFunction, argumentsOfAggregateFunction to rewrite only the agg function arguments and the grouping scalar function.
So, in the above sql, the pk + 1 AS col_alias3 will not be rewritten to slot, and can be computed.
remove vectorAnalyze and computeOutputColumn function on Expr
remove vectorOpcode and outputColumn attribute on Expr
remove useless static LOG object on some Expr class
this PR is from @sjyango work in #32326,
wants merge #32326 into master branch, but it's draft and not maintain long time. so have this new PR.
Co-authored-by: sjyango <sjyang2022@zju.edu.cn>
This PR supports a Table Value Function called `Query`. He can push a query directly to the catalog source for execution by specifying `catalog` and `query` without parsing by Doris. Doris only receives the results returned by the query.
Currently only JDBC Catalog is supported.
Example:
```
Doris > desc function query('catalog' = 'mysql','query' = 'select count(*) as cnt from test.test');
+-------+--------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------+------+------+---------+-------+
| cnt | BIGINT | Yes | true | NULL | NONE |
+-------+--------+------+------+---------+-------+
Doris > select * from query('catalog' = 'mysql','query' = 'select count(*) as cnt from test.test');
+----------+
| cnt |
+----------+
| 30000000 |
+----------+
```
when create MTMV,we will save current ctl and db.
when refresh MTMV,will create an ConnectContext, and set same ctl, db to ctx
when db,ctx dropped, task will be failed.
But sometimes deleting a db does not actually have an impact, so changing it to not directly fail. If refreshing the data does cause an error, then giving the user an error message
this pr is a quick solution, but not complete.
runtime filter on NestLoopJoin suffers this bug even without PROJECT_OTHER_JOIN_CONDITION rule.
for example, when enable Min_Max Runtime filter, the target Expression is n_regionkey, but it should be "n_regionkey - 28"
explain
select n_nationkey, nrkey
from (select n_regionkey -28 nrkey, n_nationkey from nation) T
join region on nrkey > r_regionkey;
we will refactor RuntimeFilterGenerator to completely solve this issue in following pr.
we do derive stats job eager to avoid un derive stats due to merge group and optimize group
consider:
we have two groups burned by order: G1 and G2
then we have job by order derive G2, optimize group expression in G2,
derive G1, optimize group expression in G1
if G1 merged into G2, then we maybe generated job optimize group G2 before derive G1
in this case, we will do get stats from G1's child before derive G1's child stats
then we will meet NPE in CostModel.
