Now FE does not record the update time of hms tbl's partitons, so the sql cache may be hit even the hive table's partitions have changed. This pr add a field to record the partition update time, and use it when enable sql-cache.
The cache will be missed if any partition has changed at hive side.
Use System.currentTimeMillis() but not the event time of hms event because we would better keep the same measurement with the schemaUpdateTime of external table. Add this value to ExternalObjectLog and let slave FEs replay it because it is better to keep the same value with all FEs, so the sql-cache can be hit by the querys through different FEs.
now create table use auto create partition:
AUTO PARTITION BY RANGE date_trunc(event_day, 'day')
so the value of event_day will be insert into partition of date_trunc(event_day, 'day'),
eg: select * from partition_range where date_trunc(event_day,"day")= "2023-08-07 11:00:00";
we can prune some partitions by invoke function of date_trunc("2023-08-07 11:00:00","day" );
metadata tvf list:
- backends
- catalogs
- frontends
- frontends_disks
- group_commit
- iceberg_meta
- workload_groups
fix group_commit bugs
- throw NPE when properties do not contain 'table_id'
- throw NPE when table_id's table do not exist
- throw class Cast failed when table_id's table's type is not OLAP
When the rowCount exceeds a certain threshold, refrain from generating a broadcast join.
Only enforce the best expression in CostAndEnforce Job, rather than enforcing every expression.
Remove lower bound group pruning
this pr is aim to update for filter_by_select logic and change delete limit
only support scala type in delete statement where condition
only support column nullable and predict column support filter_by_select logic, because we can not push down non-scala type to storage layer to pack in predict column but do filter logic
Since Doris support query specific tablet only, so we don't depend on tableSample to do sample, instead use grammar: TABLET(id) to do so. In OlapAnalyzeTask, we calculate which tablets would be hit and set theirs id in it, so we could get how many rows actually queried and furthur we could get the scale up ratio here
## Proposed changes
Infer the column name when create view if the column is expression
## Further comments
expr column name infer strategy as following:
| expr | example | column name(before) | Inferred column name(if position is 2) |
| ------------- | --------------------------------------- | ------------------------------ | -------------------------------------- |
| function | dayofyear() | dayofyear() | __dayofyear_1 |
| cast | cast(1 as bigint) | CAST(1 AS BIGINT) | __cast_1 |
| anylyticExpr | min() | min() | __min_1 |
| predicate | 1 in (1,2,3,4) | 1 IN (1, 2, 3, 4) | __in_predicate_1 |
| literal | 1 or 'string_var_name' | 1 or 'string_var_name' | __literal_1 |
| arithmeticExpr | & | ... & ... | __arithmetic_expr_1 |
| identifier | a or b | a or b | a or b |
| case | CASE WHEN remark = 's' THEN 1 ELSE 2 END | CASE WHEN remark = 's' THEN 1 ELSE 2 END | __case_1 |
| window | min(timestamp) OVER (...) | min(timestamp) OVER(...) | __min_1 |
SQL for example:
```sql
CREATE VIEW v1 AS
SELECT
error_code,
1,
'string',
now(),
dayofyear(op_time),
cast (source AS BIGINT),
min(`timestamp`) OVER (
ORDER BY
op_time DESC ROWS BETWEEN UNBOUNDED PRECEDING
AND 1 FOLLOWING
),
1 > 2,
2 + 3,
1 IN (1, 2, 3, 4),
remark LIKE '%like',
CASE WHEN remark = 's' THEN 1 ELSE 2 END,
TRUE | FALSE
FROM
db_test.table_test1
```
the output column name is as following:
```
error_code
__literal_1
__literal_2
__now_3
__dayofyear_4
__cast_expr_5
__min_6
__binary_predicate_7
__arithmetic_expr_8
__in_predicate_9
__like_predicate_10
__case_expr_11
__arithmetic_expr_12
```
Push down limit-distinct through left/right outer join or cross join.
such as select t1.c1 from t1 left join t2 on t1.c1 = t2.c1 order by t1.c1 limit 1;
it is painful to align node in `explain` and node in `explain physical plan`, since they use two different sets of node IDs.
This pr makes 'explain' command use node IDs of their correspond node in 'explain physical plan'
If user has database with same name mysql, will introduce problem when doing checkpoint.
Solution:
Add check for this situation, if duplicate, exit and print log info to prevent damage of metadata;
Add fe config field: mysqldb_replace_name to make things correct if user already has mysql db.
Related pr: #23087#22868
1. To ensure compatibility with the original optimizer, expose the non-lambda signature of highorder function externally.
2. fix some bugs in toSql function in the original optimizer
