Commit Graph

6194 Commits

Author SHA1 Message Date
378ced72db [chore](Nereids) more reasonable parse select list only query (#31346) 2024-02-27 16:06:26 +08:00
481d94c3fc [feature](nereids) deal the slots that appear both in agg func and grouping sets (#31318)
this PR support slot appearing both in agg func and grouping sets.
sql like below:
select sum(a) from t group by grouping sets ((a)); 

Before this PR, Nereids throw exception like below:
col_int_undef_signed cannot both in select list and aggregate functions when using GROUPING SETS/CUBE/ROLLUP, please use union instead.

This PR removes the restriction and supports this situation.
2024-02-27 10:12:33 +08:00
dd229b77b1 [fix](inverted index)Remove the strong check for parser when creating a table with inverted index (#31391) 2024-02-27 10:12:33 +08:00
c34639245e [Improvement](executor)add remote scan thread pool (#31376)
* add remote scan thread pool

* +1
2024-02-27 10:12:33 +08:00
1127b0065a [Improment](executor)Add scanbytes/scanrows condition (#31364)
* Add scanbytes/scanrows condition

* fix reg
2024-02-27 10:12:33 +08:00
f163d56a98 [feature](function) support sequence function(alias of array_range), enhance both to handle datetimev2 (#30823) 2024-02-27 10:12:19 +08:00
3cee6c6722 [fix](function) fix unexpected be core in string search function (#31312)
Fix be core in multi_match_any/multi_search_all_positions functions.
2024-02-27 10:12:18 +08:00
5d4a2d93a6 [feature](nereids) support join with joinRelation (#30909) 2024-02-26 19:07:11 +08:00
Pxl
fcea2b964e [Chore](materialized-view) forbid create mv have calculations included outside aggregate functions (#31336)
forbid create mv have calculations included outside aggregate functions
2024-02-26 19:07:11 +08:00
b1fc0ebbe7 [improvement](iceberg/paimon)support estimate row count (#31204)
Get the number of rows evaluated for iceberg and paimon.
2024-02-26 19:07:10 +08:00
f951ca2efb [refactor](stats) Remove useless async loader code. (#31380) 2024-02-26 19:07:10 +08:00
Pxl
3b7261abe4 [Mv] check delete from column exists on mv (#31321) 2024-02-26 19:07:10 +08:00
3451cd6c23 [fix](datetime) fix hour 24 on be (#31304) 2024-02-26 19:07:10 +08:00
7a9fe5d275 [enhance](mtmv)MTMV supports Hive multi-level partitioning (#31060)
Issue Number: close #xxx

For example, the hive table is partitioned by `date` and `region`, with the following 6 partitions
```
20200101
        beijing
        shanghai
20200102
        beijing
        shanghai
20200103
        beijing
        shanghai
```

If the MTMV is partitioned by `date`, then the MTMV will have three partitions: 20200101, 202000102, 20200103

If the MTMV is partitioned by `region`, then the MTMV will have two partitions: beijing, shanghai
2024-02-25 18:08:19 +08:00
7a1caf4718 [refactor](wg) enable wg by default and init normal wg in constructor (#31373)
should always enable workload group because other operations depend on it for example MTMV, and spill to disk.
the normal workload group should be created in constructor.
2024-02-25 18:08:19 +08:00
8001f73e52 [pipelineX](file scan) Improve parallel tasks if ignore data distribution (#31328) 2024-02-24 11:45:05 +08:00
aee49adf1e [opt](compute-node) refactor compute node doc and opt some default config (#31325)
* [opt](compute-node) refactor compute node doc and opt some default config

* 1

* 1
2024-02-24 11:44:53 +08:00
dcbba9a013 fix compile 2024-02-24 08:26:57 +08:00
db58104bc3 [fix](inverted index) Fix inverted index for MOR unique table #31051 (#31354) 2024-02-23 23:10:36 +08:00
481517ac6a [fix](plan) only scan node with limit and no predicate can reduce to 1 instance (#31342)
This PR #25952 introduce a opt that if a scan node has limit and predicates, use only 1 instance to save cup and memory.
But this is wrong because we can not guarantee that the predicates can truly help to prune the data.
So I modify the logic to remove this opt.
Now, only scan node with limit and NO predicate can reduce to only 1 instance.
2024-02-23 21:09:14 +08:00
49842eecc5 [Fix](multi-catalog) Fix NPE when refreshing catalog on Slave FE. (#31335)
---------

Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>
2024-02-23 20:59:07 +08:00
b6ca76e7d4 fix routine load job throw exception after commit (#31303) 2024-02-23 20:57:03 +08:00
bb31f4adb6 [fix](mtmv)fix generate partition name illegality when partition value contains colon (#31282) 2024-02-23 19:05:20 +08:00
b5ec1e7b7d [fix](Nereids) support check authorization for view but skip check in the view (#31289)
move UserAuthentication in BindRelation, support check authorization view but skip check in the view

relate pr: #23295
2024-02-23 19:03:28 +08:00
9a40b6c978 Refactor get row count related interface, add row count cache for external table. (#31276) 2024-02-23 19:03:28 +08:00
8f77e6363a [Feature](function) Support xxhash function like murmur hash function (#31193) 2024-02-23 19:03:28 +08:00
04c295c4c2 Improve show column stats performance. (#31298) 2024-02-23 19:03:28 +08:00
0bf13525d7 [opt](cancel) Cancel get result future immediately if query is cancelled (#31228) 2024-02-23 19:03:16 +08:00
e56fe682c9 fix total task exec time is far more than actual (#31279) 2024-02-22 22:22:29 +08:00
f65876d803 [Feature](explode) support explode map type (#30151) 2024-02-22 20:08:44 +08:00
caf68f3f60 [enhancement](fe) Accelerate replay journal when meta out of date (#31161) 2024-02-22 20:06:50 +08:00
100cea16d9 [fix](nereids)show error message if use wrong alias to bind the table (#31264) 2024-02-22 20:06:50 +08:00
22efb1cb7a [fix](Nereids) not equals and hashCode should contains generate flag (#31123) 2024-02-22 19:51:20 +08:00
f1a7f9a70f [opt](Nereids)choose better column for count star (#31227) 2024-02-22 19:51:20 +08:00
64056bb96b [Fix](seq-col) Fix sequence column check fail #31252
When FE generates plans and reaches the sequence column for rule judgment, an insertion statement that should have been correctly processed fails. This failure occurs because the judgment logic for the sequence part is case-sensitive to column names: the column name is in lowercase in the create table statement, but in uppercase in the insertion statement, causing the sequence column not to be correctly identified. However, Doris itself is case-insensitive to column names. This PR fixes the issue through the use of the ignoreCase method.
2024-02-22 19:51:20 +08:00
260568db17 [update](hudi) update hudi version to 0.14.1 and compatible with flink hive catalog (#31181)
1. Update hudi version from 0.13.1 to .14.1
2. Compatible with the hudi table created by flink hive catalog
2024-02-22 19:51:20 +08:00
241a2fc25c [nereids] downgrade bucket shuffle if tablet num < instance num (#31222) 2024-02-22 19:51:20 +08:00
98c3cb825f [fix](Nereids) simplify airthmetic should not change return type (#31237) 2024-02-22 19:50:47 +08:00
7647cec509 [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception (#31248) 2024-02-22 19:50:07 +08:00
389c702a3f [improvement](hudi)Obtain partition information through HMS's API (#30962)
When a Hudi table is synchronized to HMS, the partition information is also synchronized, so even if the metastore is not enabled in the Hudi table (for example, if the Metastore is false for a Hudi table created with Flink), you can still obtain the partition information through the HMS API.
2024-02-22 19:50:07 +08:00
ef22bd3318 [fix](txn insert) Txn insert can not write to table with mv (#31167) 2024-02-22 13:01:49 +08:00
ad07dec0ed [Improve](InPredict) enhance in predict with struct type (#30840) 2024-02-22 13:01:49 +08:00
d080fe8c89 [log](query) add debug log for check query visible version (#31217) 2024-02-22 13:01:48 +08:00
70dc40ab3c [fix](audit) The time field in the audit log table is set to the millisecond level (#31096) 2024-02-22 13:01:48 +08:00
98a1b12ac6 [Enhancement] [plsql] support drop store procedure (#30966) (#31001) 2024-02-22 13:01:48 +08:00
c56cb0ac3e [Exec](RF) Support merge remote rf local first (#31067) 2024-02-22 13:01:48 +08:00
2069f9ff63 [fix](Nereids): OuterJoinAssoc reject literal condition (#31172) 2024-02-22 13:01:32 +08:00
6bbe6c9a59 [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. #31212 2024-02-21 19:18:45 +08:00
eb5e6a829a [fix](nereids)should normalize window expression by bottom project's output (#31115) 2024-02-21 19:18:45 +08:00
8fc9d80479 [compatibility](MySQL) update charset to utf8mb4, collation to utf8mb4_0900_bin (#31046)
Doris's behaviour is more like utf8mb4 and utf8mb4_0900_bin than utf8 and utf8_general_ci
2024-02-21 17:01:39 +08:00