Commit Graph

6184 Commits

Author SHA1 Message Date
f951ca2efb [refactor](stats) Remove useless async loader code. (#31380) 2024-02-26 19:07:10 +08:00
Pxl
3b7261abe4 [Mv] check delete from column exists on mv (#31321) 2024-02-26 19:07:10 +08:00
3451cd6c23 [fix](datetime) fix hour 24 on be (#31304) 2024-02-26 19:07:10 +08:00
7a9fe5d275 [enhance](mtmv)MTMV supports Hive multi-level partitioning (#31060)
Issue Number: close #xxx

For example, the hive table is partitioned by `date` and `region`, with the following 6 partitions
```
20200101
        beijing
        shanghai
20200102
        beijing
        shanghai
20200103
        beijing
        shanghai
```

If the MTMV is partitioned by `date`, then the MTMV will have three partitions: 20200101, 202000102, 20200103

If the MTMV is partitioned by `region`, then the MTMV will have two partitions: beijing, shanghai
2024-02-25 18:08:19 +08:00
7a1caf4718 [refactor](wg) enable wg by default and init normal wg in constructor (#31373)
should always enable workload group because other operations depend on it for example MTMV, and spill to disk.
the normal workload group should be created in constructor.
2024-02-25 18:08:19 +08:00
8001f73e52 [pipelineX](file scan) Improve parallel tasks if ignore data distribution (#31328) 2024-02-24 11:45:05 +08:00
aee49adf1e [opt](compute-node) refactor compute node doc and opt some default config (#31325)
* [opt](compute-node) refactor compute node doc and opt some default config

* 1

* 1
2024-02-24 11:44:53 +08:00
dcbba9a013 fix compile 2024-02-24 08:26:57 +08:00
db58104bc3 [fix](inverted index) Fix inverted index for MOR unique table #31051 (#31354) 2024-02-23 23:10:36 +08:00
481517ac6a [fix](plan) only scan node with limit and no predicate can reduce to 1 instance (#31342)
This PR #25952 introduce a opt that if a scan node has limit and predicates, use only 1 instance to save cup and memory.
But this is wrong because we can not guarantee that the predicates can truly help to prune the data.
So I modify the logic to remove this opt.
Now, only scan node with limit and NO predicate can reduce to only 1 instance.
2024-02-23 21:09:14 +08:00
49842eecc5 [Fix](multi-catalog) Fix NPE when refreshing catalog on Slave FE. (#31335)
---------

Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>
2024-02-23 20:59:07 +08:00
b6ca76e7d4 fix routine load job throw exception after commit (#31303) 2024-02-23 20:57:03 +08:00
bb31f4adb6 [fix](mtmv)fix generate partition name illegality when partition value contains colon (#31282) 2024-02-23 19:05:20 +08:00
b5ec1e7b7d [fix](Nereids) support check authorization for view but skip check in the view (#31289)
move UserAuthentication in BindRelation, support check authorization view but skip check in the view

relate pr: #23295
2024-02-23 19:03:28 +08:00
9a40b6c978 Refactor get row count related interface, add row count cache for external table. (#31276) 2024-02-23 19:03:28 +08:00
8f77e6363a [Feature](function) Support xxhash function like murmur hash function (#31193) 2024-02-23 19:03:28 +08:00
04c295c4c2 Improve show column stats performance. (#31298) 2024-02-23 19:03:28 +08:00
0bf13525d7 [opt](cancel) Cancel get result future immediately if query is cancelled (#31228) 2024-02-23 19:03:16 +08:00
e56fe682c9 fix total task exec time is far more than actual (#31279) 2024-02-22 22:22:29 +08:00
f65876d803 [Feature](explode) support explode map type (#30151) 2024-02-22 20:08:44 +08:00
caf68f3f60 [enhancement](fe) Accelerate replay journal when meta out of date (#31161) 2024-02-22 20:06:50 +08:00
100cea16d9 [fix](nereids)show error message if use wrong alias to bind the table (#31264) 2024-02-22 20:06:50 +08:00
22efb1cb7a [fix](Nereids) not equals and hashCode should contains generate flag (#31123) 2024-02-22 19:51:20 +08:00
f1a7f9a70f [opt](Nereids)choose better column for count star (#31227) 2024-02-22 19:51:20 +08:00
64056bb96b [Fix](seq-col) Fix sequence column check fail #31252
When FE generates plans and reaches the sequence column for rule judgment, an insertion statement that should have been correctly processed fails. This failure occurs because the judgment logic for the sequence part is case-sensitive to column names: the column name is in lowercase in the create table statement, but in uppercase in the insertion statement, causing the sequence column not to be correctly identified. However, Doris itself is case-insensitive to column names. This PR fixes the issue through the use of the ignoreCase method.
2024-02-22 19:51:20 +08:00
260568db17 [update](hudi) update hudi version to 0.14.1 and compatible with flink hive catalog (#31181)
1. Update hudi version from 0.13.1 to .14.1
2. Compatible with the hudi table created by flink hive catalog
2024-02-22 19:51:20 +08:00
241a2fc25c [nereids] downgrade bucket shuffle if tablet num < instance num (#31222) 2024-02-22 19:51:20 +08:00
98c3cb825f [fix](Nereids) simplify airthmetic should not change return type (#31237) 2024-02-22 19:50:47 +08:00
7647cec509 [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception (#31248) 2024-02-22 19:50:07 +08:00
389c702a3f [improvement](hudi)Obtain partition information through HMS's API (#30962)
When a Hudi table is synchronized to HMS, the partition information is also synchronized, so even if the metastore is not enabled in the Hudi table (for example, if the Metastore is false for a Hudi table created with Flink), you can still obtain the partition information through the HMS API.
2024-02-22 19:50:07 +08:00
ef22bd3318 [fix](txn insert) Txn insert can not write to table with mv (#31167) 2024-02-22 13:01:49 +08:00
ad07dec0ed [Improve](InPredict) enhance in predict with struct type (#30840) 2024-02-22 13:01:49 +08:00
d080fe8c89 [log](query) add debug log for check query visible version (#31217) 2024-02-22 13:01:48 +08:00
70dc40ab3c [fix](audit) The time field in the audit log table is set to the millisecond level (#31096) 2024-02-22 13:01:48 +08:00
98a1b12ac6 [Enhancement] [plsql] support drop store procedure (#30966) (#31001) 2024-02-22 13:01:48 +08:00
c56cb0ac3e [Exec](RF) Support merge remote rf local first (#31067) 2024-02-22 13:01:48 +08:00
2069f9ff63 [fix](Nereids): OuterJoinAssoc reject literal condition (#31172) 2024-02-22 13:01:32 +08:00
6bbe6c9a59 [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. #31212 2024-02-21 19:18:45 +08:00
eb5e6a829a [fix](nereids)should normalize window expression by bottom project's output (#31115) 2024-02-21 19:18:45 +08:00
8fc9d80479 [compatibility](MySQL) update charset to utf8mb4, collation to utf8mb4_0900_bin (#31046)
Doris's behaviour is more like utf8mb4 and utf8mb4_0900_bin than utf8 and utf8_general_ci
2024-02-21 17:01:39 +08:00
f1fbfeba2f [fix](nereids) removeRuntimeFilter() removes more than one RFs if there are different types of RF from the buildNode to the target (#31197)
this bug can be produced on tpcds 95, when set runtime filter type: min-max + bloom
2024-02-21 17:01:39 +08:00
98106ad60f [fix](nereids) fix bug of unnest subuqery with having clause (#31152)
1. run PushDownFilterThroughAggregation, PushDownFilterThroughProject and MergeFilters before subquery unnesting
2. should keep plan unchanged even left semi join's condition is true (because the right table may be empty())
3. PushDownFilterThroughProject need check if filter's input slots are all coming from project's output
2024-02-21 17:01:30 +08:00
33925a738d [improvement](schemachange) add schema change log (#31145) (#31203) 2024-02-21 14:15:02 +08:00
c27692fb3b [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog (#31153) 2024-02-21 13:53:40 +08:00
cd7230885f [fix](nereids)push more than one runtime filters into cte (#30901)
* push rf into cte, used by tpcds95
2024-02-21 13:53:32 +08:00
1e3968fe7e [fix](group_commit) Need to wait wal to be deleted when creating MaterializedView (#30956) 2024-02-21 13:53:19 +08:00
1d6dc9a5f0 [fix](catalog recycle bin) Forbid recover partition if table schame is changed #31146 2024-02-21 13:53:18 +08:00
0ab8143565 [fix](catalog recycle bin) table partition meta is error if recover partition in some case (#31125) 2024-02-21 13:52:10 +08:00
97c9d75af3 [Feature](executor)Add scan_thread_num property for workload group (#31106) 2024-02-20 16:24:05 +08:00
9251816673 [fix](Nereids): log constraint editlog in lock scope (#30630) 2024-02-20 16:24:05 +08:00