1. Support prefetch some column stats when FE booted, it would load column stats that was got updated recently according to the comment of PR #18460 from @morrySnow
2. Refactor stats cache, split histogram cache from column stats, so that we could avoid some redundant query for column statistics table,for example, update the histogram or column stats only, in the previous implementation a united cache loader would send query request to both column stats table and histogram table,
3. Extract some common logic to StatsUtil
4. Remove some useless codes in unit tests, those codes is hard to maintaince and it's not a good idea for testing the accurracy of stats estimation according to the advise from @englefly
5. Add field type restriction when create analysis tasks to avoid unnecessary failure
`Hive 3` uses the `thrift-0.9.3` package, and `Doris` uses the `thrift-0.16.0` package.
These two packages are not compatible, so we use the `hive-sahde` package to manage hive dependencies
in a unified way. This jar package renames the `thrift` class , so the problem of conflict can be resolved.
Split ExternalFileScanNode into FileQueryScanNode and FileLoadScanNode.
Remove some useless code in FileLoadScanNode.
Remove unused config item: enable_vectorized_load and enable_new_load_scan_node
1. remove TypeCoercion and CharacterLiteralTypeCoercion
2. Nereids Cast do not relay on legacy planner's analyze()
3. fix below problem in legacy planner, after this PR
a. BOOLEAN can cast to DECIMALV2 explicitly
b. compare between BOOLEAN and DATE will cast both side to DOUBLE
c. HLL cannot be implicitly cast to any other type
when group-by-keys does not contain unique column
1. with out distinct: we prefer two phase aggregate to one phase aggregate
2. with distinct: we prefer three phase aggregate to two phase aggregate
steps to repo:
1, create any catalog re; [OK]
2, switch re [OK]
3, show catalogs [OK]
4, drop catalog re [OK]
5, show catalogs [FAIL with "Current catalog is not exist, please switch catalog." ]
expect:
show catalogs should always be OK, not depends on current catalog.
if inner join implemented by NLJ, the runtime filter generation phase will be terminated and children are not be travelled. we fix it by adjust the order of travelling children and handle the node itself.
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up #18397
3. Add KW_HOSTNAME to keywords region, follow up #17329
4. Fix tvf not working with pipeline engine, follow up #18376
Introduced from #17884.
When replay catalog from image, we should not call `catalog.getProperties()`.
Because it will visit the resource mgr, but resource mgr is not replayed yet.
currently, the AnalysisException throw by matchSql will catch immediately.
however, the AnalysisException throws by checkLimitations will catch as UserException.
Sometimes, `show load profile` will only show part of the insert opertion's profile.
This is because we assume that for all load operation(including insert), there is only one fragment in the plan.
But actually, there will be more than 1 fragment in plan. eg:
`insert into tbl1 select * from tbl1 limit 1` will have 2 fragments.
This PR mainly changes:
1. modify the `show load profile`
Before: `show load profile "/queryid/taskid/instanceid";`
After: `show load profile "/queryid/taskid/fragmentid/instanceid";`
2. Modify the display of `ReadColumns` in OlapScanNode
Because for wide table, the line of `ReadColumns` may be too long for show in profile.
So I wrap it and each line contains at most 10 columns names.
3. Fix tvf not working with pipeline engine, follow up #18376
Currently if a catalog is modified externally in doris, doris is not dynamically aware of it.
So if a catalog is created with a refresh time configuration, I added a timer for it to refresh the catalog regularly.
Problem: when used nereids to generate scalarType, byteSize would be set. After switch the optimizer to planner, planner would reuse scalarType in some cases.
Fix: change byteSize setting from Plan translator to toCatalogDataType
1. The time string in the profile can be "xx s xx ms". The framework should extract time with re package to support more complicated time string
2. Add stats for sortNode and AggNode in `withChildren`
validate supported data types checks if a project node's output contains any unsupported data types like array, map, etc in nereids. So this validation should run before EliminateUnnecessaryProject rule
sql : select bitmap_empty() from d_table where true;
should always use base index instead of any mv, because the conjuncts is constant (true) and use none of the column from any mv
Currently, the session variable for Split size will not take effect after the file splits are cached.
1. This PR is to cache file for Hive Table, instead of cache file splits. And split the file every time using the current split size.
2. Use self splitter by default.
