When `skip_write_index_on_load` is turned on, users will get an error when querying for the latest data(not compacted), giving them a bad experience. And we can use `inverted_index_ram_dir_enable = true` and `inverted_index_storage_format=V2` to reduce IO and CPU consumption. So we disable it now.
1. Disable setting `skip_write_index_on_load` to `true` in create table stmt.
2. Disable setting `skip_write_index_on_load` to `true` in alter table properties stmt. You can still alter `skip_write_index_on_load` to `false`.
Co-authored-by: Luennng <luennng@gmail.com>
1. fix sql cache return old value when truncate partition
2. use expire_sql_cache_in_fe_second to control the expire time of the sql cache which in the NereidsSqlCacheManager
Decoupling the MTMV from the materialization context.
Change MaterializationContext to abstract which is the materialization desc.
It now has AsyncMaterializationContext sub class, can also has other type of MaterializationContext such as
SyncMaterializationContext and so on.
1. Forbid rollback to the old optimizer in `Export` task. Since originStmt is empty, reverting to the old optimizer when the new optimizer is enabled is meaningless.
2. Display `parallelism` in 'show export'.
3. Create an initial Map for resultAttachedInfo to avoid NullPointerException.
4. Remove the originStmt added in this PR #31020, because the `Export` statement is underlying the Outfile statement, it cannot be treated as `OriginStmt`.
In some cases, using Resultset.getxxx cannot correctly handle null data, so I use Optional to enhance the fields of JdbcFieldSchema for better type mapping when some results are null.
* Revert "[refactor](mysql result format) use new serde framework to tuple convert (#25006)"
This reverts commit e5ef0aa6d439c3f9b1f1fe5bc89c9ea6a71d4019.
* run buildall
* MORE
* FIX
Need query to view expression mapping when check the logic of hyper graph is equals or not.
Getting all expression mapping one-time may affect performance. So split the expresson to three type
JOIN_EDGE, NODE, FILTER_EDGE and get them step by step.
Enhance properties regulator checking:
(1) right bucket shuffle restriction takes effective only when either side has NATUAL shuffle type.
(2) enhance bothSideShuffleKeysAreSameOrder checking if taking EquivalenceExprIds into consideration.
Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
This PR improves the materialized view rule by adding additional validity checks for the struct info. The changes include:
1. In AbstractMaterializedViewRule.java:
* Modified the checkPattern method to also check the validity of the struct info using context.getStructInfo().isValid().
* Updated the error message for invalid struct info to include the view plan string for better debugging.
2. In StructInfo.java:
* Enhanced the isValid() method to ensure that all nodes in the hyper graph have non-null expressions, in addition to the existing validity check.
These changes ensure that the materialized view rule properly validates the struct info before proceeding with the optimization. It prevents invalid struct info from being used and provides more detailed error messages for debugging purposes.
The additional validity check in StructInfo.isValid() verifies that all nodes in the hyper graph have non-null expressions, which is necessary for the materialized view rule to function correctly.
By incorporating these validity checks, the materialized view rule becomes more robust and can handle invalid struct info gracefully, improving the overall stability and reliability of the optimization process.
when the runtime filter target to a constant, the PushDownContext.finalTarget is null.
in this case, do not pushdown runtime filter.
example:
select * from (select 1 as x from T1) T2 join T3 on T2.x=T3.x
when push down RF(T3.x->T2.x) inside "select 1 as x from T1", the column x is a constant, and the pushDown stopped.
(cherry picked from commit 7f06cf0a0125d84c41e1edbb71366e293f70239d)
Problem and Cause:
In original planner, date_add function would choose different data type of datetime or datev2. Which when original planner choose datev2 as constant date type. Fe could not folding date_add function because missing of function signature of folding constant date_add(datev2, int)
Solved:
Add corresponding function signatures of date_add/months_add/years_add in original planner
baseTables in MTMVRelation stores all baseTables in the nested materialized view,
now adding baseTablesOneLevel to only store the baseTables at the current level.
* [refactor](Nereids)refactor runtime filter generator (#34275)
1. unify the process of generating rf for hash join and for nested loop join
2. fix some bugs in generating rf
3. remove some duplicated check
(cherry picked from commit 07267faac0d9c6ef3bb1fd4ee101b4c761c8a2f2)
* [refactor](nereids) do not deny a runtime filter by removing an entry in aliasMap (#34559)
in current version, there are 2 approaches to verify whether a join condition can be used to generate a runtime filter, they are
1. remove the output slot from aliasMap
2. pushDownVisitor.visit(...) return false
the 1st approach has some drawbacks, we prefer to the 2ed approach.
In this pr, all the cases are handled by the 2ed approach, and remove the related code for the 1st approach.
(cherry picked from commit a29082bf31e66efa2df193b38347e610f2bf7464)
* rebase
