Commit Graph

17549 Commits

Author SHA1 Message Date
3e8c75e246 [minor](orc) opt the log info in orc reader (#27951) 2023-12-06 20:47:36 +08:00
ec08850c08 [Config](multi-catalog) Enable query hive views as default. (#27906)
Remove EXPERIMENTAL tag for enable_query_hive_views and set enable_query_hive_views to true as default.
This feature has been used on our cluster which has more then a hundred thousands of tables for several months, i think it is fine to enable it as default.
2023-12-06 20:46:09 +08:00
3868926088 [fix](ci) pass PULL_REQUEST_NUM while trigger new build (#28065)
Co-authored-by: stephen <hello-stephen@qq.com>
2023-12-06 20:42:53 +08:00
Pxl
299fcc443e [Bug](agg-state) fix stream load failed on agg-state column (#28050) 2023-12-06 20:41:29 +08:00
990b8d4fa5 [minor](s3client) use LOG(INFO) to print s3 client debug log if enabled (#28059) 2023-12-06 20:39:12 +08:00
Pxl
5168422da7 [Chore](case) remove dynamic partition on test_delete case #28075
remove dynamic partition on test_delete case to reduce logout
2023-12-06 16:23:12,733 WARN (DynamicPartitionScheduler|58) [DynamicPartitionScheduler.recordCreatePartitionFailedMsg():581] dynamic add partition failed: errCode = 2, detailMessage = Range [types: [DATETIMEV2]; keys: [0000-01-01 00:00:00]; ..types: [DATETIMEV2]; keys: [2021-01-01 00:00:00]; ) is intersected with range: [types: [DATETIMEV2]; keys: [2020-03-01 00:00:00]; ..types: [DATETIMEV2]; keys: [2020-04-01 00:00:00]; ), db: default_cluster:regression_test_delete_p0, table: dwd_pay
2023-12-06 20:35:08 +08:00
ff56605c2e [pipelineX](bug) Fix hash partition shuffle (#28071) 2023-12-06 19:26:46 +08:00
ffd7023987 [feature](nereids) Support to get partition related table from mv and check the query operator (#28064)
Function 1:
check the select query plan is contain the stmt as following or not

SELECT
[hint_statement, ...]
[ALL | DISTINCT | DISTINCTROW | ALL EXCEPT ( col_name1 [, col_name2, col_name3, ...] )]
elect_expr [, select_expr ...]
[FROM table_references
PARTITION partition_list]
[TABLET tabletid_list]
[TABLESAMPLE sample_value [ROWS | PERCENT]
[REPEATABLE pos_seek]]
[WHERE where_condition]
[GROUP BY [GROUPING SETS | ROLLUP | CUBE] {col_name | expr | position}]
[HAVING where_condition]
[ORDER BY {col_name | expr | position}
[ASC | DESC], ...]
[LIMIT {[offset,] row_count | row_count OFFSET offset}]
[INTO OUTFILE 'file_name']

if analyzedPlan contains the stmt as following

[PARTITION partition_list]
[TABLET tabletid_list] or
[TABLESAMPLE sample_value [ROWS | PERCENT]
[REPEATABLE pos_seek]]

this method will return true.

Function 2:
Get related base table info which materialized view plan column reference,
input param plan should be rewritten plan that sub query should be eliminated
2023-12-06 19:15:21 +08:00
e695cb18b1 [improvement](regression test) add fe debug point (#28047) 2023-12-06 18:18:05 +08:00
17be32eefa [fix](regression test) fix table name too long (#28049) 2023-12-06 18:16:17 +08:00
ddb6eb5ad7 [feature](Nereids) add command for updating mv with partitions (#28060) 2023-12-06 17:45:09 +08:00
1aa1b2f607 [opt](Nereids) add switch to control whether use pipeline in DML (#28037)
to turn on pipeline for DML in Nerieds, please:
set enable_nereids_dml_with_pipeline = true;
2023-12-06 17:06:11 +08:00
bb67e39558 [fix](invert index) fix reader is not close (#28033) 2023-12-06 17:02:26 +08:00
28817990b7 [pipelineX](improvement) enable local shuffle by default (#28046) 2023-12-06 16:39:48 +08:00
fa5096f510 [feature](analyze_cmd) add show-tablets-belong stmt for analyzing a batch of tablet-ids (#27994) 2023-12-06 15:59:00 +08:00
994c5c6f6e [chore](log) Add log to trace query cancel #28020 2023-12-06 15:51:21 +08:00
605257ccb7 [Enhancement](group commit) Add regression case for wal limit (#27949) 2023-12-06 14:23:50 +08:00
e791e31b7f [test](Nereids): add regression test eliminate/infer rules (#27985) 2023-12-06 14:21:06 +08:00
393c491820 [FIX](map/struct)fix map/struct literal from fe (#28026) 2023-12-06 13:56:56 +08:00
0a22d969e1 [refactor](queryqueue) using a priority queue in query queue in order to implement priority management in the future (#27969) 2023-12-06 13:49:11 +08:00
a0fee4c96e [fix](nereids) runtime filter prune skip filter with invisible column (#28010)
if a conjunct only contains invisible column, this conjunct should not be used in runtime filter pruner
2023-12-06 12:42:40 +08:00
24fdb7ad4e [refactor](unusedcode) remove internalquery since it is useless (#28039)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-12-06 12:30:49 +08:00
cbf1f8620a [Feature](job)support cancel task and fix log invalid (#27703)
- Running task can be show and fix cancel fail
- When the insert task scheduling cycle is reached, if there are still tasks running, the scheduling of this task will be canceled at this time.
- refactor job status changes SQL
- Fix timer job window error
- Support cancel task
2023-12-06 10:44:09 +08:00
4c90b459e5 [regression-test](variant) fix variant p2 cases (#28025) 2023-12-06 10:37:18 +08:00
202cc84f2f [Test](auto-partition) Add auto partition p1 test (#28028)
Add auto partition p1 test
2023-12-06 10:20:39 +08:00
1be513b927 [pipelineX](local shuffle) Fix local shuffle for colocate/bucket join (#28032) 2023-12-06 10:02:36 +08:00
73404405e6 [fix](hdfs) Fix hdfsExists that return staled root cause (#27991)
The HDFS native client won't clear the last exception as expected so `hdfsGetLastExceptionRootCause` might return a staled root cause. This PR saves the last root cause here and verifies after hdfsExists returns a non-zero code.
2023-12-06 08:24:13 +08:00
57fb1df833 [fix](regression) backup restore cases adapt force_olap_table_replica_num option (#27986) 2023-12-06 08:23:40 +08:00
36c54da03d fix owned slice capacity (#28002) 2023-12-06 00:26:34 +08:00
aaa44fa3dc [deps](odbc) undefined the BOOL in include/sqltypes.h (#28014)
When `ENABLE_PCH = false`, this define will be conflict with BOOL in `include/arrow/type_fwd.h`.
The ODBC table will be deprecated in 2.1, so I just simply undefined the BOOL in include/sqltypes.h
to make compile OK
2023-12-06 00:16:23 +08:00
e431c2b980 [Improvement](multi-catalog)make location easier to modified, decoupling all storage with single location class (#27874)
decoupling all storage with single location class
2023-12-06 00:13:54 +08:00
fd1db4da3d [agg](profile) fix incorrent profile (#28004) 2023-12-05 20:48:10 +08:00
7f1b558011 [fix](stats) truncate min/max if too long (#27955)
For some string value the max/min might be a very long string
which might take too many memory of FE,
so we truncate to 1024 chars if it's too long
2023-12-05 20:40:38 +08:00
05adbfdb3d [feature](inverted index) match_phrase_prefix feature added (#27404)
select count() from test_index_match_phrase_prefix where request match_phrase_prefix 'xxx';
2023-12-05 20:15:13 +08:00
ffa4ea66d5 [enhancement](main) donot coredump when be can not start (#27928) 2023-12-05 20:11:24 +08:00
ea275e687a [pipelineX](minor) remove unused code (#28016) 2023-12-05 19:41:40 +08:00
e79422addc [refactor](Nereids) compatible with all ability legacy planner (#27947)
refactor:
1. split InsertIntoTableCommand into three sub command
- InsertIntoTableCommand
- InsertOverwriteTableCommand
- BatchInsertIntoTableCommand

feature:
1. support DEFAULT keywords in values list
2. support empty values list
3. support temporary partition
4. support insert into values in txn model

fix:
1. should start transaction before release read lock on target table
2023-12-05 19:10:55 +08:00
8e161ad0f2 [debug](timeout) add some log to debug timeout== 0 (#28011)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-12-05 18:35:36 +08:00
59c0db4080 Fix workload group unstable (#28003) 2023-12-05 17:55:43 +08:00
8dc67d3be6 [case](regression-test) add backup recovery for inverted index and bl… (#27616) 2023-12-05 17:11:02 +08:00
6074cddcf8 [feature](mtmv)add Job and task tvf (#27967)
add:
select * from jobs("type"="mv");
select * from tasks("type"="mv");
select * from jobs("type"="insert");
select * from tasks("type"="insert");

add check priv for mv_infos("database"="xxx");

change JobType MTMV==>MV
2023-12-05 15:12:36 +08:00
02512cd0e2 [fix](stats)Drop stats or update updated rows after truncate table (#27931)
1. Also clear follower's stats cache when doing drop stats.
2. Drop stats when truncate a table.
2023-12-05 14:53:35 +08:00
Pxl
8a761dff84 [Bug](materialized-view) fix create mv failed on unique table (#27971)
fix create mv failed on unique table
2023-12-05 14:53:09 +08:00
3595f21405 [improvement](executor)clear unused cgroup path (#27798)
* clear unused cgroup path

* use C++ api

* add gcc header
2023-12-05 14:18:23 +08:00
c98b80ae6a [Feature](functions) support ignore and nullable functions (#27848)
support ignore and nullable functions
2023-12-05 14:09:32 +08:00
79f6f85cf1 [FIX](serde)fix datetimev2 serde parse from string with scale (#27965) 2023-12-05 13:58:32 +08:00
aa75f81af5 [fix](case) add "nonConcurrent" to prevent effect other case (#27941)
same as #27940
2023-12-05 13:04:25 +08:00
54fe1a166b [Refactor](scan) refactor scan scheduler to improve performance (#27948)
* [Refactor](scan) refactor scan scheduler to improve performance

* fix pipeline x core
2023-12-05 13:03:16 +08:00
da87fcb477 [bug](function) fix compound expr coredump problem (#27988) 2023-12-05 13:00:14 +08:00
fa0b495b33 [fix](cache)Fix partition cache support DATEV2 (#27978) 2023-12-05 12:59:47 +08:00