1. Implement needReAnalyzeTable for ExternalTable. For now, external table will not be reanalyzed in 10 days.
2. For HiveMetastoreCache.loadPartitions, handle the empty iterator case to avoid Index out of boundary exception.
3. Wrap handle show analyze loop with try catch, so that when one table failed (for example, catalog dropped so the table couldn't be found anymore), we can still show the other tables.
4. For now, only OlapTable and Hive HMSExternalTable support sample analyze, throw exception for other types of table.
5. In StatisticsCollector, call constructJob after createTableLevelTaskForExternalTable to avoid NPE.
The old logic used to determine whether it was a partition table based on the number of buckets, but if I had a partition table with only one partition and the number of buckets in that partition was 1, it would be mistakenly recognized as a non partition table.
```
Table[test_load_doris_to_hive_2] is not partitioned
```
Use the cached information and estimated information of the table in the rows column under
information_schema.tables. Avoid querying information_schema.tables that will cause rpc timeout when there are a
large number of tables in the catalog.
Support where, group by, having, order by clause without from clause in query statement.
For example as following:
SELECT 1 AS a, COUNT(), SUM(2), AVG(1), RANK() OVER() AS w_rank
WHERE 1 = 1
GROUP BY a, w_rank
HAVING COUNT() IN (1, 2) AND w_rank = 1
ORDER BY a;
this will return result:
| a |count(*)|sum(2)|avg(1)|w_rank|
+----+--------+------+------+------+
| 1 | 1| 2| 1.0| 1|
For another example as following:
select 1 c1, 2 union (select "hell0", "") order by c1
the second column datatype will be varchar(65533), 65533 is the default varchar length.
this will return result:
|c1 | 2 |
+------+---+
|1 | 2 |
|hell0 | |
materialized view rewrite framework, support to query rewrite by struct info.
The idea is from "Optimizing Queries Using Materialized Views- A Practical, Scalable Solution"
Currently doris doesn't support datetime type filter stats estimation, but only for date type.
It will cause the filter using datetime type column with the same date and different time computing out a inaccurate selectivity and estimate a wrong row count, such as :
where o.book_time >= '2020-03-01 00:00:00.0' and o.book_time <= '2020-03-01 23:59:59.0';
This pr adds the datetime type(only support hh:mm:ss scale) filter estimation and improve the row count estimation for the above case.
Fix auto analyze doesn't filter unsupported type bug.
Catch throwable in auto analyze thread for each database, otherwise the thread will quit when one database failed to create jobs and all other databases will not get analyzed.
change FE config item full_auto_analyze_simultaneously_running_task_num to auto_analyze_simultaneously_running_task_num
The creation of hudi and iceberg table is disallowed since v1.2.
All these features are covered by hudi/iceberg catalog.
We should remove the code in v2.1
The PR mainly changes:
1. remove the code of hudi/iceberg external table.
2. remove code of iceberg database.
3. disallowed hive external table's creation.
4. disabled odbc,mysql,broker external table by default, and add FE config `disable_odbc_mysql_broker_table` to control it
Introduction to Main Classes：
- MTMVService：MTMV services for other modules to call
- MTMVHookService：All operations that affect the MTMV
- MTMVJobManager：All operations that affect the MTMV job
- MTMVCacheManager：All operations that affect the MTMV Cache
- MTMVTask&MTMVJob：Inherit from job framework
