Commit Graph

4151 Commits

Author SHA1 Message Date
ea2fbfaffa [feature](Nereids) support agg state type in create table (#32171)
this PR introduce a behavior change, syntax of create table with agg_state type is changed.
2024-03-15 18:04:49 +08:00
c919834df3 [fix](test) ckbench shape test failed because parallel merge (#32224)
query36's shape broken by PR #32186
2024-03-15 18:03:03 +08:00
4534300030 [fix](Operator) RepeatNode does not handle empty expressions. (#32112)
In the past, RepeatNode did not handle empty expressions.
It used DCHECK to check if the expression was non-empty.
In non-debug mode, this caused _child_block to remain unprocessed, resulting in a deadlock.
Now, if the expression is empty, the output block directly outputs _child_block
2024-03-15 18:02:33 +08:00
7205f267cc [opt](Nereids) support cast agg state type as legacy planner (#32198) 2024-03-15 18:02:01 +08:00
680fce8825 [chore](test) let regression test work well with nereids (#32199) 2024-03-15 18:02:01 +08:00
9150c82545 [test](Nereids) add ckbench shape test (#32191) 2024-03-15 18:02:01 +08:00
36a0b93c44 [Enhancement](mor) Add unique mor table min max push down case #32196 2024-03-15 18:02:01 +08:00
bede948029 [enhancement](nereids) support unnest subquery with group by and having clause (#32002) 2024-03-15 18:01:49 +08:00
3f4ae002a8 [fix](merge-cloud) fix no cluster for common user (#32097) 2024-03-15 18:00:42 +08:00
3c4234111b [fix](nereids)EliminateSemiJoin should consider empty table (#32107)
* [fix](nereids)EliminateSemiJoin should consider empty table

* update out file
2024-03-15 17:59:31 +08:00
c8c6e86386 [fix](Nereids): ignore project and distribute in test_cte_filter_pushdown and push_down_expression_in_hash_join (#32083) 2024-03-15 17:59:30 +08:00
99830b77a8 [feat](nereids) add merge aggregate rule (#31811) 2024-03-15 17:58:01 +08:00
fd1e0e933e [opt](Nereids) outer join with is null stats estimation enhancement (#31875) 2024-03-15 17:58:01 +08:00
94a75c27e7 [feature](pipelineX) support paritition tablet sink shuffle (#31689) 2024-03-15 17:58:01 +08:00
df5ec16d7c [Refactor](exectuor)Add schema type table active_queries (#32057)
* Add schema type table active_queries
2024-03-15 17:57:28 +08:00
5f75b36ad3 [regression](framework) add config caseNamePrefix (#32266) 2024-03-14 21:48:28 +08:00
b031c95324 [Opt](exec) use libbase64 to replace base64 code in doris (#32078)
* [Opt](exec) use libbase64 to replace base64 code in doris
2024-03-14 09:20:50 +08:00
84af8e0a53 [enhance](mtmv)mtmv support hive default partition (#32051) 2024-03-12 22:51:11 +08:00
6acdd9cd48 The issue introduced by the recently added auto-increment column works fine on a single node but may result in discontinuous auto-increment IDs when running on a cluster. This PR has been modified to check for the uniqueness of the auto-increment column values instead of checking for equality to a fixed value. (#32115) 2024-03-12 21:51:36 +08:00
473bd3ee64 [fix](function) incorrect result of eq_for_null (#32103) 2024-03-12 18:50:26 +08:00
4956d5de83 [fix](planner) remove input slot for aggregate slot which is not materialized (#32092)
introduced by #26886

run this sql:
SELECT
        caseId
    FROM
        (
            SELECT
                caseId,
                count(judgementDateId)
            FROM
                (
                    SELECT
                        abs(caseId) AS caseId,
                        id as judgementDateId
                    FROM
                        dr_user_test_t2
                ) AGG_RESULT
            GROUP BY
                caseId
        ) TOTAL
        order by 1;


will get:

ERROR 1105 (HY000): errCode = 2, detailMessage = (172.17.0.1)[INTERNAL_ERROR]couldn't resolve slot descriptor 1, desc: tuples:
Tuple(id=5 slots=[Slot(id=10 type=DOUBLE col=-1, colname=, nullable=1), Slot(id=11 type=VARCHAR col=-1, colname=id, nullable=1)] has_varlen_slots=1)
Tuple(id=4 slots=[Slot(id=8 type=DOUBLE col=-1, colname=, nullable=1)] has_varlen_slots=0)
Tuple(id=2 slots=[Slot(id=4 type=DOUBLE col=-1, colname=caseId, nullable=1)] has_varlen_slots=0)
Tuple(id=0 slots=[Slot(id=0 type=VARCHAR col=-1, colname=caseId, nu
2024-03-12 18:50:26 +08:00
781a45d93c [Fix](nereids) fix date function rewrite (#32060) 2024-03-12 18:50:26 +08:00
cf04c9c300 [enhancement](Nereids) refine and speedup analyzer (#31792) (#32111)
## Proposed changes
1. check data type whether can applied should not throw exception when real data type is subclass of signature data type
2. merge `SlotBinder` and `FunctionBinder` to `ExpressionAnalyzer` to skip rewrite the whole expression tree multiple times.
3. `ExpressionAnalyzer.buildCustomSlotBinderAnalyzer()` provide more refined code to bind slot by different parts and different priority
4. the origin slot binder has O(n^2) complexity, this pr use `Scope.nameToSlot` to support O(n) bind
5. modify some `Collection.stream()` to `ImmutableXxx.builder()` to remove some method call which are difficult to inline by jvm in the hot path, e.g. `Expression.<init>` and `AbstractTreeNode.<init>`
6. modify some `ImmutableXxx.copyOf(xxx)` to `Utils.fastToImmutableList(xxx)` to skip addition copy of the array
7. set init size to `Immutable.builder()` to skip some useless resize
8. lazy compute and cache some heavy operations, like `Scope.nameToSlot` and `CaseWhen.computeDataTypesForCoercion()`

(cherry picked from commit 83c2f5a95827136aac4f0a78c5e841e9a099858c)
2024-03-12 17:09:38 +08:00
194f3432ab [Improvement](executor)Routine load support workload group #31671 2024-03-12 14:20:18 +08:00
dc7d80860f [fix](case) fix export data consistency table key type (#32045) 2024-03-12 14:20:18 +08:00
ccd21a6ea4 [Improve](InPredict) enhance in predict with array type (#31828) 2024-03-12 14:19:14 +08:00
a7a85dd330 [chore](config) support select experimental session variable (#31837)
support select experimental variables:

Before change:


Before change:
select @@experimental_enable_nereids_planner;
ERROR 1193 (HY000): errCode = 2, detailMessage = Unknown system variable 'experimental_enable_nereids_planner'

show variables like 'experimental_enable_nereids_planner';
+-------------------------------------+-------+---------------+---------+
| Variable_name                       | Value | Default_Value | Changed |
+-------------------------------------+-------+---------------+---------+
| experimental_enable_nereids_planner | false | true          | 1       |
+-------------------------------------+-------+---------------+---------+

After change:
> select @@experimental_enable_nereids_planner;
+---------------------------------------+
| @@experimental_enable_nereids_planner |
+---------------------------------------+
|                                     1 |
+---------------------------------------+
2024-03-12 14:18:26 +08:00
31ee448c87 [test](fix) Fix one missing line of output in out file (#32036) 2024-03-12 14:17:55 +08:00
ab21d85e8c [nereids](topn-filter) support multi-topn filter (FE part) (#31485)
support multi-topn-filter
2024-03-12 14:17:48 +08:00
cf6b22c621 [fix](jdbc catalog) fix type conversion error in MySQL JDBC Driver 5.x (#31880) 2024-03-12 14:07:57 +08:00
27eed5399d [Fix](auto-inc) Fix partial update auto inc publish case failure #31987 2024-03-12 14:07:00 +08:00
c5390d00bb [Improvement]Add schema table backend_active_tasks (#31945) 2024-03-09 19:55:48 +08:00
d5bf20c96e [improvement](mtmv) Improve the performance for query rewritting by materialized view (#31886)
- Limit the number of times for the query rewritting to the group
- Remove the unnecessary log and explain detail info in query
2024-03-09 19:55:47 +08:00
1d094a46ec [regression-test](pipeline) remove sys_log_verbose_modules in pipeline #32015 2024-03-09 19:55:47 +08:00
263135c193 [fix](case) fix export data consistency case (#32005) 2024-03-09 19:45:50 +08:00
21e412393b [enhancement] add_method_for_schemachange (#31849) 2024-03-09 19:45:50 +08:00
aaa5542fb9 fix drop un-related table (#31990) 2024-03-09 19:45:50 +08:00
cc0e58faec [enhancement](regression-test) upgrade groovy to 4.x and enable run test by jdk17/21 (#31906)
upgrade groovy to 4.x and enable run test by jdk17 / 21
2024-03-09 19:45:46 +08:00
5f9eb5eb52 [feature](external catalog)Add partition grammar for external catalog to create table (#31585)
The `PARTITION BY` syntax used by external catalogs has been added. 
You can specify a column directly, or a partition function as a partition condition.
Like:
 `PARTITION BY LIST(col1, col2, func(param), func(param1, param2), func(param1, param2, param3))`

NOTICE:
This PR change the grammar of `AUTO PARTITION`
From 
```
AUTO PARTITION BY RANGE date_trunc(`TIME_STAMP`, 'month')
```
To
```
AUTO PARTITION BY RANGE (date_trunc(`TIME_STAMP`, 'month'))
```
2024-03-09 19:45:46 +08:00
62db7094ea Revert "Problem: When the old optimizer processes an INSERT INTO statement that contains two quotation marks, it results in only one quotation mark being written into the database. (#31890)" (#31986)
This reverts commit 8c309652e04698f311b6c9158105352e8416c69a.
2024-03-09 19:45:46 +08:00
5909237ab1 [Fix](nereids) Add semantic check that the hash bucket column must be a key column when creating table for aggregate and unique models (#31951) 2024-03-09 19:45:46 +08:00
609761567c [Fix](partial-update) Fix wrong column number passing to BE when partial and enable nereids (#31461)
* Problem:
Inconsistent behavior occurs when executing partial column update `UPDATE` statements and `INSERT` statements on merge-on-write tables with the Nereids optimizer enabled. The number of columns passed to BE differs; `UPDATE` operations incorrectly pass all columns, while `INSERT` operations correctly pass only the updated columns.

Reason:
The Nereids optimizer does not handle partial column update `UPDATE` statements properly. The processing logic for `UPDATE` statements rewrites them as equivalent `INSERT` statements, which are then processed according to the logic of `INSERT` statements. For example, assuming a MoW table structure with columns k1, k2, v1, v2, the correct rewrite should be:
* `UPDATE` table t1 set v1 = v1 + 1 where k1 = 1 and k2 = 2
 * =>
 * `INSERT` into table (v1) select v1 + 1 from table t1 where k1 = 1 and k2 = 2

However, the actual rewriting process does not consider the logic for partial column updates, leading to all columns being included in the `INSERT` statement, i.e., the result is:
* `INSERT` into table (k1, k2, v1, v2) select k1, k2, v1 + 1, v2 from table t1 where k1 = 1 and k2 = 2

This results in `UPDATE` operations incorrectly passing all columns to BE.

Solution:
Having analyzed the cause, the solution is straightforward: when rewriting partial column update `UPDATE` statements to `INSERT` statements, only retain the updated columns and all key columns (as partial column updates must include all key columns). Additionally, this PR includes error injection cases to verify the number of columns passed to BE is correct.

* 2

* 3

* 4

* 5
2024-03-09 19:45:42 +08:00
e8aa5ee7d5 [Improve](Variant) support bloom filter for variant subcolumns (#31347)
* [Improve](Variant) support bloom filter for variant subcolumns

* rebase
2024-03-09 19:45:03 +08:00
Pxl
19e6ebd09c [Feature](materialized-view) support mv with bitmap_union(bitmap_from_array()) case (#31962)
support mv with bitmap_union(bitmap_from_array()) case
2024-03-09 19:45:03 +08:00
1b783aaa7f [fix](p2)Fix analyze hive partition column p2 case after row count change. #31958 2024-03-09 19:45:03 +08:00
eb280d374b [case](Nereids) add leading tpc-h (#30405)
add tpc-h shape cases using leading hint
except:

single table without join q1 q6
not support feature include tables after subquery unnested q2 q16 q18 q20 q21 q22
2024-03-09 19:45:03 +08:00
93d298d34a [fix](agg) wrong result of two or more map_agg functions in query (#31928) 2024-03-09 19:45:03 +08:00
e9c1638507 Add waiting timeout while creating mv and row count report. (#31944) 2024-03-09 19:44:54 +08:00
aff09fc9bc [feature](Nereids) support make miss slot as null alias when converting anti join (#31854)
transform

project(A.*, B.slot)
  - filter(B.slot is null)
    - LeftOuterJoin(A, B)

to

project(A.*, null as B.slot)
  - LeftAntiJoin(A, B)
2024-03-09 19:43:21 +08:00
8801916675 [regression](spark)Add spark to read doris multiple data types cases (#31861) 2024-03-09 19:43:21 +08:00