Commit Graph

8289 Commits

Author SHA1 Message Date
65fc2e0438 [fix](Nereids) forbid two TVF in one fragment since the limit of coordinator (#28114) 2023-12-07 19:58:31 +08:00
cc9b4bcddb [Fix](variant) fallback to none partial update for mow table (#28116) 2023-12-07 19:30:24 +08:00
34642781c2 [fix](meta) fix ConcurrentModificationException when dump image (#28072)
```
Caused by: java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) ~[?:1.8.0_131]
        at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_131]
        at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_131]
        at org.apache.doris.catalog.CatalogRecycleBin.write(CatalogRecycleBin.java:1047) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.saveRecycleBin(Env.java:2298) ~[doris-fe.jar:1.2-SNAPSHOT]
```

When calling `/dump` api to dump image, ConcurrentModificationException may be thrown.
Because no lock to protect `CatalogRecycleBin`
2023-12-07 18:26:02 +08:00
394b420180 [Update](inverted index) use session variable for inverted index try query threshold (#28052)
* [Update](inverted index) use session variable for inverted index try query threshold

* remove unused config

* update clucene
2023-12-07 17:54:44 +08:00
bc12a05915 [fix](Nereids) explain graph insert-select NPE (#28007) 2023-12-07 17:25:44 +08:00
495c01bdfd [fix](session-variable) enable_unicode_name_support need forward to master (#28112)
Session variable `enable_unicode_name_support` is used to enable using unicode for table/column names.
And it should be forward to master when executing `create table` stmt on non-master FE.
Otherwise, even if we set `enable_unicode_name_support` to true on non-master FE,
we can create table with unicode name.
2023-12-07 15:43:11 +08:00
8526b9ffbe [imporvement](table property) support for alter table property disable_auto_compaction (#27961)
in some case, some tablets may cause coredump or OOM when compaction, and it is necessary to manually close the compaction of a specific table by 'disable_auto_compaction' to make be service available

This commit allow modify disable_auto_compaction table property in schema change.

---------

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-12-07 15:08:39 +08:00
8c79b86f5b Revert "[feature](merge-on-write) enable merge-on-write by default (#27188)" (#28096)
This reverts commit 00c8bab84de8154052f9d323800b436cd0ad36e5.
2023-12-07 11:31:36 +08:00
3a7a8bb107 [opt](resource-tag) root and admin user can use any resource tag by default (#28088)
In #25331, I change the behavior of user's default resource tag, that is, if a user does not set resource tag,
it can only use default resource tag.
This PR change this logic. The normal user can only use default resource tag if resource tag is not set,
but root and admin user can use any resource tag if resource tag is not set.
2023-12-07 11:22:30 +08:00
b6722653cf [test](Job)Delete the JOB show syntax (now we use TVF) and add tvf case (#28058) 2023-12-07 10:17:52 +08:00
630de740ea [fix](meta) fix bug for using full name in show_full_columns stmt (#28019) 2023-12-07 10:17:25 +08:00
4cac07be30 [improvement](statistics)Analyze empty table. #28077
Analyze a table even when it's empty. The result should be like this:

mysql> show column stats nation;
+-------------+-------+------+----------+-----------+---------------+------+------+--------+--------------+---------+-------------+---------------------+
| column_name | count | ndv  | num_null | data_size | avg_size_byte | min  | max  | method | type         | trigger | query_times | updated_time        |
+-------------+-------+------+----------+-----------+---------------+------+------+--------+--------------+---------+-------------+---------------------+
| n_comment   | 0.0   | 0.0  | 0.0      | 0.0       | 0.0           | N/A  | N/A  | FULL   | FUNDAMENTALS | MANUAL  | 0           | 2023-12-06 19:22:09 |
| n_nationkey | 0.0   | 0.0  | 0.0      | 0.0       | 0.0           | N/A  | N/A  | FULL   | FUNDAMENTALS | MANUAL  | 0           | 2023-12-06 19:22:09 |
| n_regionkey | 0.0   | 0.0  | 0.0      | 0.0       | 0.0           | N/A  | N/A  | FULL   | FUNDAMENTALS | MANUAL  | 0           | 2023-12-06 19:22:09 |
| n_name      | 0.0   | 0.0  | 0.0      | 0.0       | 0.0           | N/A  | N/A  | FULL   | FUNDAMENTALS | MANUAL  | 0           | 2023-12-06 19:22:09 |
+-------------+-------+------+----------+-----------+---------------+------+------+--------+--------------+---------+----
2023-12-07 10:16:52 +08:00
42b3dd35bb [regression test](broker load) add case for without filepath (#27658) 2023-12-07 10:15:37 +08:00
54d062ddee [feature](stream load) (step one)Add arrow data type for stream load (#26709)
By using the Arrow data format, we can reduce the streamload of data transferred and improve the data import performance
2023-12-06 23:29:46 +08:00
4a4d137402 [feature](workloadgroup) support nereids internal query and all dml query (#28054)
support nereids internal query to bind a workload group
support insert into select bind workload group
support create table as select bind workload group
change token wait timeout to be query timeout or queue timeout
query queue should not bind to pipeline engine, it could be used every where.
2023-12-06 21:07:55 +08:00
00c8bab84d [feature](merge-on-write) enable merge-on-write by default (#27188) 2023-12-06 21:06:58 +08:00
0ff5a1cc25 [fix](doc) spell error and aligned with code (#27609) 2023-12-06 20:58:39 +08:00
ec08850c08 [Config](multi-catalog) Enable query hive views as default. (#27906)
Remove EXPERIMENTAL tag for enable_query_hive_views and set enable_query_hive_views to true as default.
This feature has been used on our cluster which has more then a hundred thousands of tables for several months, i think it is fine to enable it as default.
2023-12-06 20:46:09 +08:00
Pxl
299fcc443e [Bug](agg-state) fix stream load failed on agg-state column (#28050) 2023-12-06 20:41:29 +08:00
ffd7023987 [feature](nereids) Support to get partition related table from mv and check the query operator (#28064)
Function 1:
check the select query plan is contain the stmt as following or not

SELECT
[hint_statement, ...]
[ALL | DISTINCT | DISTINCTROW | ALL EXCEPT ( col_name1 [, col_name2, col_name3, ...] )]
elect_expr [, select_expr ...]
[FROM table_references
PARTITION partition_list]
[TABLET tabletid_list]
[TABLESAMPLE sample_value [ROWS | PERCENT]
[REPEATABLE pos_seek]]
[WHERE where_condition]
[GROUP BY [GROUPING SETS | ROLLUP | CUBE] {col_name | expr | position}]
[HAVING where_condition]
[ORDER BY {col_name | expr | position}
[ASC | DESC], ...]
[LIMIT {[offset,] row_count | row_count OFFSET offset}]
[INTO OUTFILE 'file_name']

if analyzedPlan contains the stmt as following

[PARTITION partition_list]
[TABLET tabletid_list] or
[TABLESAMPLE sample_value [ROWS | PERCENT]
[REPEATABLE pos_seek]]

this method will return true.

Function 2:
Get related base table info which materialized view plan column reference,
input param plan should be rewritten plan that sub query should be eliminated
2023-12-06 19:15:21 +08:00
ddb6eb5ad7 [feature](Nereids) add command for updating mv with partitions (#28060) 2023-12-06 17:45:09 +08:00
1aa1b2f607 [opt](Nereids) add switch to control whether use pipeline in DML (#28037)
to turn on pipeline for DML in Nerieds, please:
set enable_nereids_dml_with_pipeline = true;
2023-12-06 17:06:11 +08:00
28817990b7 [pipelineX](improvement) enable local shuffle by default (#28046) 2023-12-06 16:39:48 +08:00
fa5096f510 [feature](analyze_cmd) add show-tablets-belong stmt for analyzing a batch of tablet-ids (#27994) 2023-12-06 15:59:00 +08:00
994c5c6f6e [chore](log) Add log to trace query cancel #28020 2023-12-06 15:51:21 +08:00
e791e31b7f [test](Nereids): add regression test eliminate/infer rules (#27985) 2023-12-06 14:21:06 +08:00
393c491820 [FIX](map/struct)fix map/struct literal from fe (#28026) 2023-12-06 13:56:56 +08:00
0a22d969e1 [refactor](queryqueue) using a priority queue in query queue in order to implement priority management in the future (#27969) 2023-12-06 13:49:11 +08:00
a0fee4c96e [fix](nereids) runtime filter prune skip filter with invisible column (#28010)
if a conjunct only contains invisible column, this conjunct should not be used in runtime filter pruner
2023-12-06 12:42:40 +08:00
24fdb7ad4e [refactor](unusedcode) remove internalquery since it is useless (#28039)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-12-06 12:30:49 +08:00
cbf1f8620a [Feature](job)support cancel task and fix log invalid (#27703)
- Running task can be show and fix cancel fail
- When the insert task scheduling cycle is reached, if there are still tasks running, the scheduling of this task will be canceled at this time.
- refactor job status changes SQL
- Fix timer job window error
- Support cancel task
2023-12-06 10:44:09 +08:00
1be513b927 [pipelineX](local shuffle) Fix local shuffle for colocate/bucket join (#28032) 2023-12-06 10:02:36 +08:00
e431c2b980 [Improvement](multi-catalog)make location easier to modified, decoupling all storage with single location class (#27874)
decoupling all storage with single location class
2023-12-06 00:13:54 +08:00
7f1b558011 [fix](stats) truncate min/max if too long (#27955)
For some string value the max/min might be a very long string
which might take too many memory of FE,
so we truncate to 1024 chars if it's too long
2023-12-05 20:40:38 +08:00
05adbfdb3d [feature](inverted index) match_phrase_prefix feature added (#27404)
select count() from test_index_match_phrase_prefix where request match_phrase_prefix 'xxx';
2023-12-05 20:15:13 +08:00
e79422addc [refactor](Nereids) compatible with all ability legacy planner (#27947)
refactor:
1. split InsertIntoTableCommand into three sub command
- InsertIntoTableCommand
- InsertOverwriteTableCommand
- BatchInsertIntoTableCommand

feature:
1. support DEFAULT keywords in values list
2. support empty values list
3. support temporary partition
4. support insert into values in txn model

fix:
1. should start transaction before release read lock on target table
2023-12-05 19:10:55 +08:00
8e161ad0f2 [debug](timeout) add some log to debug timeout== 0 (#28011)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-12-05 18:35:36 +08:00
6074cddcf8 [feature](mtmv)add Job and task tvf (#27967)
add:
select * from jobs("type"="mv");
select * from tasks("type"="mv");
select * from jobs("type"="insert");
select * from tasks("type"="insert");

add check priv for mv_infos("database"="xxx");

change JobType MTMV==>MV
2023-12-05 15:12:36 +08:00
02512cd0e2 [fix](stats)Drop stats or update updated rows after truncate table (#27931)
1. Also clear follower's stats cache when doing drop stats.
2. Drop stats when truncate a table.
2023-12-05 14:53:35 +08:00
Pxl
8a761dff84 [Bug](materialized-view) fix create mv failed on unique table (#27971)
fix create mv failed on unique table
2023-12-05 14:53:09 +08:00
c98b80ae6a [Feature](functions) support ignore and nullable functions (#27848)
support ignore and nullable functions
2023-12-05 14:09:32 +08:00
54fe1a166b [Refactor](scan) refactor scan scheduler to improve performance (#27948)
* [Refactor](scan) refactor scan scheduler to improve performance

* fix pipeline x core
2023-12-05 13:03:16 +08:00
fa0b495b33 [fix](cache)Fix partition cache support DATEV2 (#27978) 2023-12-05 12:59:47 +08:00
17016b9797 [improvement](decimal) use new way for decimal arithmetic precision promotion (#27787)
* [DNM](decimal) use new way for decimal arithmetic precision promotion

* [improvement](decimal) [DNM](decimal) use new way for decimal arithmetic precision promotion
1. [DNM](decimal) use new way for decimal arithmetic precision promotion
2. throw exception if it overflows for decimal arithmetics
3. throw exception if it overflows when casting among number types

* fix compile error of gcc

* improvement

---------

Co-authored-by: morrySnow <morrysnow@126.com>
2023-12-05 12:54:40 +08:00
ca6949ee3e [Bug](partition) fix auto list partition erros of incorrect partition name (#27974)
the partition name need limit it's length and can't have negative "-"
2023-12-05 12:54:06 +08:00
2f63999066 [fix](Nereids): Preserve "" in single quote strings and '' in double quote strings. (#27959) 2023-12-05 12:30:03 +08:00
da40e1c767 [feature](nereids) Matiarilzed view query rewrite util implementation (#27568)
The basic util implementatation which is used by materialized view rewrite
2023-12-05 11:48:04 +08:00
26d642d5e9 [enhancement](Nereids) format some code in functional deps (#27797) 2023-12-05 11:45:03 +08:00
4afe07e12c [feature](Nereids): support drop constraint on table (#27944) 2023-12-05 11:41:25 +08:00
3412a022f4 [fix](restore) fix Restore from __keep_on_local__ throws null pointer… (#26943)
Co-authored-by: walter <patricknicholas@foxmail.com>
Co-authored-by: hugoluo <hugoluo@tencent.com>
Co-authored-by: walter <patricknicholas@foxmail.com>
2023-12-05 10:55:28 +08:00