Commit Graph

7018 Commits

Author SHA1 Message Date
3cb0deae9c [opt](ranger) modify and enhance the feature of ranger access controller (#34392) (#34426)
bp #34392
2024-05-06 17:08:47 +08:00
7ae5de316b [feature](Nereids) support set and use statement syntax only (#33979) (#34409)
pick from master #33979
commit id 65fb7d43b7e838c48502d4e8a69e2541dc73aa88

This PR:
1. add a new Command type: UnsupportedCommand to handle the statement only support parse but could not execute.
2. support syntax about set and use
3. add keyword VAULT to follow legacy planner

TODO
1. support all statment syntax in Nereids
2024-05-06 11:36:01 +08:00
7248420cfd [chore](session_variable) Add 'data_queue_max_blocks' to prevent the DataQueue from occupying too much memory. (#34017) (#34395) 2024-05-05 21:20:33 +08:00
c3096cabe2 [Fix](executor)normal group not auth #34377 2024-05-02 15:17:19 +08:00
8da260ee0d [fix](hdfs)read 'fs.defaultFS' from core-site.xml for hdfs load which has no default fs (#34217) (#34372)
bp #34217
Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
2024-05-01 00:31:49 +08:00
581e168ee1 [Fix](executor)Fix normal workload group alter may failed #34356 2024-04-30 22:17:40 +08:00
5fc1f11cf1 [improvement](hive)add the queryid to the temporary file path (#34278) (#34368)
bp #34278

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-04-30 22:00:05 +08:00
50855f01c7 [fix](nereids) when runtimefilter target is null, skip the rf #34358 2024-04-30 18:48:50 +08:00
35f8563a75 [feature](iceberg) support iceberg equality delete (#34223) (#34327)
bp #34223

Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
2024-04-30 11:51:29 +08:00
d1df0b8878 [fix](mtmv)Solving the problem of calling each other in toString() loops (#34277) (#34317)
bp #34277
2024-04-29 21:46:29 +08:00
3495ed58e0 [Enhancement](jdbc catalog) Change Jdbc connection pool to hikari (#34045) (#34310) 2024-04-29 20:22:48 +08:00
7cb00a8e54 [Feature](hive-writer) Implements s3 file committer. (#34307)
Backport #33937.
2024-04-29 19:56:49 +08:00
1bfe0f0393 [feature](iceberg)support read iceberg complex type,iceberg.orc format and position delete. (#33935) (#34256)
master #33935
2024-04-29 14:40:12 +08:00
11039ade7b [opt](paimon) support mapping Paimon column type "Row" to Doris type "Struct" (#34239)
backport: #33786
2024-04-28 19:38:50 +08:00
1fda68f738 [feature](planner) Support select constant from dual syntax sugar (#34200) (#34232)
In MySQL, it's common to use a simplified syntax like `SELECT constant FROM dual`
which is equivalent to just `SELECT constant`.
This syntax is often used by BI tools when utilizing MySQL connectors to verify connection validity.
To enhance compatibility and ensure seamless integration with such tools,
we have now implemented this feature in Doris.

### Key Changes:
- Doris now interprets `SELECT constant FROM dual` as `SELECT constant`, aligning with MySQL's behavior.
- This update ensures that BI tools can use standard MySQL connectors without modifications or errors when connecting to Doris.
2024-04-28 15:56:16 +08:00
341f5cd7a3 [fix](branch-2.1) Fix streamload profile not set (#34221) 2024-04-28 14:36:58 +08:00
7ab425ee4b [improve](move-memtable) reduce default load stream per node to 2 for stream load (#34065) (#34205)
Co-authored-by: Kaijie Chen <ckj@apache.org>
2024-04-27 18:20:57 +08:00
36e80af327 [fix](schema change) fix the defineName field is not the same when copying column (#34201)
* [fix](schema change) fix the defineName field is not the same when copying column

* fix
2024-04-27 11:59:07 +08:00
4b9772062b [refactor](statistic) fetch statistic data with catalog and database id (#33928) (#34202)
bp #33928
2024-04-27 09:38:41 +08:00
c998e2f714 [Enhancement](planner) Support string input for sql_select_limit (#34177) 2024-04-27 02:29:47 +08:00
414fbd353e [fix](ES catalog)Make col != '' behavior consistent with SQL (#34151)
In SQL syntax, `col != ''` equals `col.length() > 0`.
It means that this column must exist in ES doc fields and its content is not empty.
In this PR, we make a special translation for this binary predicate to keep the behavior of both consistent.

---------

Co-authored-by: Luennng <luennng@gmail.com>
2024-04-27 02:29:33 +08:00
3ba42a7823 [improvement](mtmv) Optimize the nested materialized view performance (#34163)
Record increase refersh version more accurately.
The refreshVersion in the memo will increase when mv rewrite successfully.
In query rewrite, if refresh version is different from the current struct info map in group,
will refresh the group struct info or not.
2024-04-27 02:29:33 +08:00
c125148deb [opt](Nereids) bucket shuffle downgrade expansion (#34088)
Expand bucket shuffle downgrade condition, which originally requiring a single partition after pruning, basic table and bucket number < para number. Currently, we expect this option can be used for disabling bucket shuffle more efficiently, without above restrictions.

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
2024-04-27 02:29:33 +08:00
5e9eb417ad [bugfix](insert) fix cherry pick : redundant branch judgment #34160 2024-04-27 02:19:39 +08:00
90040e7f92 [Fix] fix compile problem (#34184)
fix compile problem in branch-2.1
2024-04-26 17:57:16 +08:00
627245f93a [fix](Nereids) support not in predicate for delete command (#34153) 2024-04-26 15:06:28 +08:00
91887a285e Implement HLL with 128 buckets to support statistics cache. (#34124) 2024-04-26 15:05:36 +08:00
b24ff9953d [fix](Nereids) column pruning should prune map in cte consumer (#34079)
we save bi-map in cte consumer to get the maping between producer and consumer.
the consumer's output is decided by the map in it.
so, cte consumer should be output prunable, and should remove useless entry from map when do column pruning
2024-04-26 15:05:19 +08:00
b41a5339d3 [Fix](nereids) fix rule merge_aggregate when has project (#33892) 2024-04-26 15:05:09 +08:00
a34ed4643a [fix](planner)date_add function should accept date type as its param (#34035) 2024-04-26 15:04:45 +08:00
5adc823b14 [fix](nereids)move ReplaceVariableByLiteral rule to analyze phase (#33997) 2024-04-26 15:04:45 +08:00
b7b87fbb95 [fix](planner)cast expr should do nothing in compactForLiteral method (#34047) 2024-04-26 15:04:45 +08:00
fdf91759b6 [fix](nereids)prevent null pointer access if translate expression fails (#33990) 2024-04-26 15:04:33 +08:00
7af5fc7321 [fix](Nereids) check after rewrite cannot handle agg in other opeator (#34114)
this is a stupid mistake. we import a same name class from another package
2024-04-26 07:43:40 +08:00
52031c86b7 [improvement](mtmv) Optimize the performance of nested materialized view rewriting (#34127)
Optimize the performance of nested materialized view rewriting gracefully, future performance optimzie base on this.
2024-04-26 07:43:25 +08:00
75644392f4 [fix](Nereids) support aggregate function only in having statement (#34086)
SQL like

> SELECT 1 AS c1 FROM t HAVING count(1) > 0 OR c1 IS NOT NULL
2024-04-26 07:41:45 +08:00
a237f7ec6e [feature](Nereids): add equal set in functional dependencies (#33642) 2024-04-26 07:41:45 +08:00
50e9982729 [Enhancement](external catalog) Added status reset when jdbc name mapping is abnormal (#34129) 2024-04-25 20:40:25 +08:00
b873be6588 [fix](routine-load) fix routine load lag is negative (#34113)
* [fix](routine-load) fix routine load lag is negative (#33846)

* fix merge error
2024-04-25 17:24:41 +08:00
987f755206 [Fix](nereids) fix rule SimplifyWindowExpression (#34099)
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2024-04-25 15:07:09 +08:00
c2e3defe56 [fix](tablet invert index) fix tablet invert index leaky caused by auto partition (#33973) 2024-04-25 15:06:43 +08:00
789a16ec6b [fix](fe) Fix SHOW CREATE TABLE with AUTO PARTITION (#34071)
AUTO PARTITION grammar has changed since #31585, but the output
of SHOW CREATE TABLE was left out to change, so the result is not
able to be recognized by the FE parser.
2024-04-25 15:05:58 +08:00
47b54d4bd5 Fix remote scan pool (#33976) 2024-04-25 15:04:43 +08:00
5f2d0e3d53 [Fix](executor)Fix when Fe send empty wg list to be may cause query failed. (#34074) 2024-04-25 12:01:44 +08:00
450f443413 [fix](decommission) fix cann't decommission mtmv (#33823) 2024-04-25 12:01:44 +08:00
a15a8e119f [fix](mtmv) Fix exception when create materialized view with cte (#33988)
Fix exception when create materialized view with cte, after this fix, can create materialized view with following
```
        CREATE MATERIALIZED VIEW mv_with_cte
            BUILD IMMEDIATE REFRESH AUTO ON MANUAL
            DISTRIBUTED BY RANDOM BUCKETS 2
            PROPERTIES ('replication_num' = '1')
            AS
            with `test_with` AS (
            select l_partkey, l_suppkey
            from lineitem
            union
            select
              ps_partkey, ps_suppkey
            from
            partsupp)
            select * from test_with;
```

this is brought from https://github.com/apache/doris/pull/28144
2024-04-25 12:01:44 +08:00
eaacba644d [fix](auth)can not grant priv to __internal_schema (#34009)
mysql> grant SELECT_PRIV on `_internal_schema`.* to 'test'@'%'; ERROR 1102 (42000): errCode = 2, detailMessage = Incorrect database name '_internal_schema'
2024-04-25 12:01:44 +08:00
103987ebd8 [fix](parse) set origin stmt for select stmt generated from show stmt (#34015)
* [fix](parse) set origin stmt for select stmt generated from show stmt

* 2
2024-04-25 12:01:44 +08:00
fbd2c9db2d [upgrade](hive-shade)(paimon) upgrade hive shade to 2.0.0 and paimon to 0.7 (#34085)
* Adapt paimon 0.6.0 (#33943)

Version 2.0.0 of the shade package eliminates potential jar conflicts, resolves dependency component issues, and significantly reduces package size.
Utilize the directly-dependent guava library instead of relying on transitively included libraries.

* [chore](dependencies)Upgrade paimon to 0.7.0 (#33987)

---------

Co-authored-by: Calvin Kirs <kirs@apache.org>
2024-04-25 12:01:44 +08:00
7eb35c95f3 [refactor](errormessage) step1: unify the status usage in FE (#34062)
We should tell the user the correct error message when some thing wrong. But error message is in a mess. I will make it clear.

This is the first step: unify the error code usage in FE.
2024-04-25 12:01:44 +08:00