Commit Graph

15360 Commits

Author SHA1 Message Date
42b3dd35bb [regression test](broker load) add case for without filepath (#27658) 2023-12-07 10:15:37 +08:00
e3e78a79d7 [chore](StoragePolicy) Don't print stack when invoking the storage manager #28091 2023-12-07 10:06:31 +08:00
54d062ddee [feature](stream load) (step one)Add arrow data type for stream load (#26709)
By using the Arrow data format, we can reduce the streamload of data transferred and improve the data import performance
2023-12-06 23:29:46 +08:00
2ca66ff61c [typo](test) Use correct sql stmt in regression test (#27992) 2023-12-06 23:28:29 +08:00
4a4d137402 [feature](workloadgroup) support nereids internal query and all dml query (#28054)
support nereids internal query to bind a workload group
support insert into select bind workload group
support create table as select bind workload group
change token wait timeout to be query timeout or queue timeout
query queue should not bind to pipeline engine, it could be used every where.
2023-12-06 21:07:55 +08:00
00c8bab84d [feature](merge-on-write) enable merge-on-write by default (#27188) 2023-12-06 21:06:58 +08:00
d0697db7b8 [fix](doris compose) fix up command (#27573) 2023-12-06 21:06:10 +08:00
d4f9e12ec7 [fix](load) fix memtable mem_tracker too large (#25205) 2023-12-06 21:04:21 +08:00
0ff5a1cc25 [fix](doc) spell error and aligned with code (#27609) 2023-12-06 20:58:39 +08:00
bd808ae3b4 [cleanup](load) remove unused LoadChannel::get_tablets_channels (#27981) 2023-12-06 20:50:26 +08:00
0282baaf56 [typo](docker) add logs console (#27570)
doris 2.0.2+ need use start_xx.sh --console
2023-12-06 20:50:09 +08:00
3e8c75e246 [minor](orc) opt the log info in orc reader (#27951) 2023-12-06 20:47:36 +08:00
ec08850c08 [Config](multi-catalog) Enable query hive views as default. (#27906)
Remove EXPERIMENTAL tag for enable_query_hive_views and set enable_query_hive_views to true as default.
This feature has been used on our cluster which has more then a hundred thousands of tables for several months, i think it is fine to enable it as default.
2023-12-06 20:46:09 +08:00
3868926088 [fix](ci) pass PULL_REQUEST_NUM while trigger new build (#28065)
Co-authored-by: stephen <hello-stephen@qq.com>
2023-12-06 20:42:53 +08:00
Pxl
299fcc443e [Bug](agg-state) fix stream load failed on agg-state column (#28050) 2023-12-06 20:41:29 +08:00
990b8d4fa5 [minor](s3client) use LOG(INFO) to print s3 client debug log if enabled (#28059) 2023-12-06 20:39:12 +08:00
Pxl
5168422da7 [Chore](case) remove dynamic partition on test_delete case #28075
remove dynamic partition on test_delete case to reduce logout
2023-12-06 16:23:12,733 WARN (DynamicPartitionScheduler|58) [DynamicPartitionScheduler.recordCreatePartitionFailedMsg():581] dynamic add partition failed: errCode = 2, detailMessage = Range [types: [DATETIMEV2]; keys: [0000-01-01 00:00:00]; ..types: [DATETIMEV2]; keys: [2021-01-01 00:00:00]; ) is intersected with range: [types: [DATETIMEV2]; keys: [2020-03-01 00:00:00]; ..types: [DATETIMEV2]; keys: [2020-04-01 00:00:00]; ), db: default_cluster:regression_test_delete_p0, table: dwd_pay
2023-12-06 20:35:08 +08:00
ff56605c2e [pipelineX](bug) Fix hash partition shuffle (#28071) 2023-12-06 19:26:46 +08:00
ffd7023987 [feature](nereids) Support to get partition related table from mv and check the query operator (#28064)
Function 1:
check the select query plan is contain the stmt as following or not

SELECT
[hint_statement, ...]
[ALL | DISTINCT | DISTINCTROW | ALL EXCEPT ( col_name1 [, col_name2, col_name3, ...] )]
elect_expr [, select_expr ...]
[FROM table_references
PARTITION partition_list]
[TABLET tabletid_list]
[TABLESAMPLE sample_value [ROWS | PERCENT]
[REPEATABLE pos_seek]]
[WHERE where_condition]
[GROUP BY [GROUPING SETS | ROLLUP | CUBE] {col_name | expr | position}]
[HAVING where_condition]
[ORDER BY {col_name | expr | position}
[ASC | DESC], ...]
[LIMIT {[offset,] row_count | row_count OFFSET offset}]
[INTO OUTFILE 'file_name']

if analyzedPlan contains the stmt as following

[PARTITION partition_list]
[TABLET tabletid_list] or
[TABLESAMPLE sample_value [ROWS | PERCENT]
[REPEATABLE pos_seek]]

this method will return true.

Function 2:
Get related base table info which materialized view plan column reference,
input param plan should be rewritten plan that sub query should be eliminated
2023-12-06 19:15:21 +08:00
e695cb18b1 [improvement](regression test) add fe debug point (#28047) 2023-12-06 18:18:05 +08:00
17be32eefa [fix](regression test) fix table name too long (#28049) 2023-12-06 18:16:17 +08:00
ddb6eb5ad7 [feature](Nereids) add command for updating mv with partitions (#28060) 2023-12-06 17:45:09 +08:00
1aa1b2f607 [opt](Nereids) add switch to control whether use pipeline in DML (#28037)
to turn on pipeline for DML in Nerieds, please:
set enable_nereids_dml_with_pipeline = true;
2023-12-06 17:06:11 +08:00
bb67e39558 [fix](invert index) fix reader is not close (#28033) 2023-12-06 17:02:26 +08:00
28817990b7 [pipelineX](improvement) enable local shuffle by default (#28046) 2023-12-06 16:39:48 +08:00
fa5096f510 [feature](analyze_cmd) add show-tablets-belong stmt for analyzing a batch of tablet-ids (#27994) 2023-12-06 15:59:00 +08:00
994c5c6f6e [chore](log) Add log to trace query cancel #28020 2023-12-06 15:51:21 +08:00
605257ccb7 [Enhancement](group commit) Add regression case for wal limit (#27949) 2023-12-06 14:23:50 +08:00
e791e31b7f [test](Nereids): add regression test eliminate/infer rules (#27985) 2023-12-06 14:21:06 +08:00
393c491820 [FIX](map/struct)fix map/struct literal from fe (#28026) 2023-12-06 13:56:56 +08:00
0a22d969e1 [refactor](queryqueue) using a priority queue in query queue in order to implement priority management in the future (#27969) 2023-12-06 13:49:11 +08:00
a0fee4c96e [fix](nereids) runtime filter prune skip filter with invisible column (#28010)
if a conjunct only contains invisible column, this conjunct should not be used in runtime filter pruner
2023-12-06 12:42:40 +08:00
24fdb7ad4e [refactor](unusedcode) remove internalquery since it is useless (#28039)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-12-06 12:30:49 +08:00
cbf1f8620a [Feature](job)support cancel task and fix log invalid (#27703)
- Running task can be show and fix cancel fail
- When the insert task scheduling cycle is reached, if there are still tasks running, the scheduling of this task will be canceled at this time.
- refactor job status changes SQL
- Fix timer job window error
- Support cancel task
2023-12-06 10:44:09 +08:00
4c90b459e5 [regression-test](variant) fix variant p2 cases (#28025) 2023-12-06 10:37:18 +08:00
202cc84f2f [Test](auto-partition) Add auto partition p1 test (#28028)
Add auto partition p1 test
2023-12-06 10:20:39 +08:00
1be513b927 [pipelineX](local shuffle) Fix local shuffle for colocate/bucket join (#28032) 2023-12-06 10:02:36 +08:00
73404405e6 [fix](hdfs) Fix hdfsExists that return staled root cause (#27991)
The HDFS native client won't clear the last exception as expected so `hdfsGetLastExceptionRootCause` might return a staled root cause. This PR saves the last root cause here and verifies after hdfsExists returns a non-zero code.
2023-12-06 08:24:13 +08:00
57fb1df833 [fix](regression) backup restore cases adapt force_olap_table_replica_num option (#27986) 2023-12-06 08:23:40 +08:00
36c54da03d fix owned slice capacity (#28002) 2023-12-06 00:26:34 +08:00
aaa44fa3dc [deps](odbc) undefined the BOOL in include/sqltypes.h (#28014)
When `ENABLE_PCH = false`, this define will be conflict with BOOL in `include/arrow/type_fwd.h`.
The ODBC table will be deprecated in 2.1, so I just simply undefined the BOOL in include/sqltypes.h
to make compile OK
2023-12-06 00:16:23 +08:00
e431c2b980 [Improvement](multi-catalog)make location easier to modified, decoupling all storage with single location class (#27874)
decoupling all storage with single location class
2023-12-06 00:13:54 +08:00
fd1db4da3d [agg](profile) fix incorrent profile (#28004) 2023-12-05 20:48:10 +08:00
7f1b558011 [fix](stats) truncate min/max if too long (#27955)
For some string value the max/min might be a very long string
which might take too many memory of FE,
so we truncate to 1024 chars if it's too long
2023-12-05 20:40:38 +08:00
05adbfdb3d [feature](inverted index) match_phrase_prefix feature added (#27404)
select count() from test_index_match_phrase_prefix where request match_phrase_prefix 'xxx';
2023-12-05 20:15:13 +08:00
ffa4ea66d5 [enhancement](main) donot coredump when be can not start (#27928) 2023-12-05 20:11:24 +08:00
ea275e687a [pipelineX](minor) remove unused code (#28016) 2023-12-05 19:41:40 +08:00
e79422addc [refactor](Nereids) compatible with all ability legacy planner (#27947)
refactor:
1. split InsertIntoTableCommand into three sub command
- InsertIntoTableCommand
- InsertOverwriteTableCommand
- BatchInsertIntoTableCommand

feature:
1. support DEFAULT keywords in values list
2. support empty values list
3. support temporary partition
4. support insert into values in txn model

fix:
1. should start transaction before release read lock on target table
2023-12-05 19:10:55 +08:00
8e161ad0f2 [debug](timeout) add some log to debug timeout== 0 (#28011)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-12-05 18:35:36 +08:00
59c0db4080 Fix workload group unstable (#28003) 2023-12-05 17:55:43 +08:00