Commit Graph

6274 Commits

Author SHA1 Message Date
d94d2c65f6 [fix](Nereids) let OrToIn rewritten result have stable order (#31731) 2024-03-06 13:07:59 +08:00
2d6e975d5a [fix](cast) fix wrong result while cast string to float (#31781)
Issue Number: close #31518
2024-03-06 13:07:59 +08:00
eea9b56f69 [fix](group commit) handle group commit create plan error (#31757) 2024-03-06 13:07:59 +08:00
cc8112273e [fix](mtmv) Avoiding the occurrence of null pointers in logs due to the deletion of MTMV (#31722) 2024-03-06 13:07:49 +08:00
97640ee0e8 [test](leading) add leading tpc-ds regression test cases (#31681)
Co-authored-by: libinfeng <libinfeng@selectdb.com>
2024-03-06 13:07:49 +08:00
d9943d0374 [chore](Nereids) change consumer rules name (#31777) 2024-03-06 13:07:49 +08:00
ad3308c8ab [fix](hive) support partition prune for _HIVE_DEFAULT_PARTITION_ (#31736)
This PR #23026 support the partition prune for hive table with `_HIVE_DEFAULT_PARTITION`,
but it will always select partition with `_HIVE_DEFAULT_PARTITION`.

This PR #31613 support null partition for olap table's list partition, so we can treat `_HIVE_DEFAULT_PARTITION`
as null partition of hive table.

So this PR change the partition prune logic
2024-03-06 13:07:49 +08:00
ca59579bf6 fix](Nereids): support max_value in range partition (#31721) 2024-03-06 13:07:08 +08:00
7998da4691 [fix](cast) wrong result while cast const to double then to string (#31657)
Issue Number: close #31514
2024-03-06 13:06:27 +08:00
2c26a308d1 [fix](broker-load) fix broker load statement type conversion failure (#31746)
Co-authored-by: Luwei <814383175@qq.com>
2024-03-06 13:06:27 +08:00
7c30cb20fd [Fix](partial update) Fix partial update load false when schema includes auto increment column (#31725)
Problem:
When partially updating columns without specifying the auto-increment column, and the imported data contains new keys, an error stating the auto-increment column could not be found occurs.

Reason:
The logic for partial column updates does not account for new keys in auto-increment columns. Since auto-increment columns can be generated by the system, it's possible to omit this column data during import. However, partial column updates treat this as a regular column, expecting it to be nullable or have a default value for automatic filling, overlooking the fact that auto-increment columns can also be auto-filled. This oversight leads to the error.

Solution:
Incorporate a check for auto-increment columns into the partial column update logic, and include the logic for generating auto-increment column values in the process of completing partial updates.
2024-03-06 13:06:27 +08:00
d7d8b4ee4a [fix](fe) Fix NPE in FrontendServiceImpl.loadTxnCommit if table is dropped (#31715) 2024-03-06 13:05:31 +08:00
a26b1eeee7 [fix](connections) fix connection hang after too many connections (#31594)
Issue Number: close #31569

Fix fe connection hang after too high qps

After fix, the third SQL will return error instead of hang:
ERROR 1203 (HY000): #42000Too many connections
2024-03-06 13:05:22 +08:00
Pxl
0c71b666d7 [Bug](resource) fix npe on Resource read from json (#31723)
fix npe on Resource read from json
2024-03-06 13:05:22 +08:00
3777ffb43f [enhancement](nereids)support null partition for list partition (#31613) 2024-03-06 13:05:22 +08:00
33c356e6d8 fix in stats estimation (#31684)
the valid in option should

! (op.min > expr.max or op.max < expr.min) 
=============> 
op.min <= expr.max and op.max >= expr.min
2024-03-06 13:04:54 +08:00
e2ebf9d566 [feature](Nereids) parallel output file (#31623)
legacy planner impl PR: #6539
2024-03-06 13:04:30 +08:00
8198a31563 [fix](testcases) make all auto partition P2 cases nonConcurrent (#31708) (#31821)
make all auto partition P2 cases nonConcurrent
2024-03-06 09:12:02 +08:00
bc898e3cdc [pipelineX](bug) Fix incorrect join operator judgement (#31690)
* [pipelineX](bug) Fix incorrect join operator judgement

* update
2024-03-03 19:22:41 +08:00
47d330c74d [enhancement](Nereids) New optimizer support check column privileges (#31700)
* [enhancement](Nereids) New optimizer support check column privileges (#31494)

(cherry picked from commit accfcfcae5a3343a9e2b7241453e56738255de1b)

* column privilege

* column privilege
2024-03-03 13:48:31 +08:00
de9b5f7b69 [improvement](statistics)Log one bdbje record for one load transaction. #31619 (#31697) 2024-03-02 23:04:26 +08:00
cacbd048ec [fix](arrow-flight)Fix the prompt that be arrow_flight_sql_port is not set (#31668) 2024-03-02 17:12:49 +08:00
874f4c693b [feat](nereids) support nullSafeEqual estimation (#31616) 2024-03-01 19:07:00 +08:00
38cb17567a [opt](plsql) Fix plsql exception and doris exception compatibility (#31647) 2024-03-01 19:07:00 +08:00
b26dcf2677 [opt](nereids) Composite predicate supports range predicate when rewritting by materialzied view (#31538)
It supports predicate composite as following:
materialized view define
>        select l_shipdate, o_orderdate, l_partkey, l_suppkey
>        from lineitem_1
>        left join orders_1
>        on lineitem_1.l_orderkey = orders_1.o_orderkey
>        where l_shipdate > '2023-10-19'


the query as following can be rewritten by the materialized view above
>        select l_shipdate, o_orderdate, l_partkey, l_suppkey
>        from lineitem_1
>        left join orders_1
>        on lineitem_1.l_orderkey = orders_1.o_orderkey
>        where l_shipdate > '2023-10-25'
2024-03-01 19:06:59 +08:00
493c9d49ea [fix](Nereids) check slot from children may produce wrong format err (#31638) 2024-03-01 17:00:13 +08:00
535e3b23ea [Fix](nereids) fix leading with left join without leftside condition (#31387)
When left join with no edge in outer side, we should add outer side to minimal require of left tables


Co-authored-by: libinfeng <libinfeng@selectdb.com>
2024-03-01 16:59:53 +08:00
a8e2652f78 [fix](planner) should not push down sink when last exchange do limit (#31622) 2024-03-01 14:21:36 +08:00
Pxl
ba07acadc2 [Bug](delete) ignore case on delete from command and add check on deletejob dispatch (#31593)
ignore case on delete from command and add check on deletejob dispatch
2024-03-01 14:19:28 +08:00
07224686ef [feature](jdbc catalog) support db2 jdbc catalog (#31627) 2024-03-01 14:19:28 +08:00
0d77fc4847 [nereids] explode function stats configurable (#31332)
---------

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
2024-03-01 04:25:43 +08:00
07703633dd [pipelineX](conf) enable pipelineX by default (#29894)
* update

* update
2024-03-01 04:25:43 +08:00
2047b9416f [Fix](nereids) Only rewrite the slots that appear both in trival-agg func and grouping sets (#31600)
* [Fix](nereids) Only rewrite the slots that appear both in trival-agg func and grouping sets

* [Fix](nereids) Only rewrite the slots that appear both in trival-agg func and grouping sets

---------

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2024-03-01 04:25:43 +08:00
1f825ee2d6 [improve](export) Support partition data consistency (#31290) 2024-03-01 04:25:43 +08:00
82faa7469b Support analyze rollup. (#31576) 2024-03-01 04:25:43 +08:00
819ab6fc00 [feature](sink) support paritition tablet sink shuffle (#30914)
Co-authored-by: morrySnow <morrysnow@126.com>
2024-03-01 04:25:43 +08:00
54709ecf3b [improvement](plsql) Select statement supports insert into variables #31574 2024-02-29 19:51:18 +08:00
92e3b31f50 [feature](invert index) match_phrase_edge feature added (#31142) 2024-02-29 19:51:18 +08:00
de28d7cd2d [fix](Nereids): AssertNumRowsElement shouldn't be expression (#31581) 2024-02-29 19:51:07 +08:00
720b6e3d86 [fix](nereids) forbid create table with illegal auto partition expr (#31498) (#31604) 2024-02-29 18:08:37 +08:00
01e6798049 [fix](Neredis) Fix npe when plan node doesn't have expressions in materialized view (#31579) 2024-02-29 16:44:40 +08:00
9988d6f9fb [Fix](executor)Fix insert select forward carry workload_group #31578 2024-02-29 16:44:40 +08:00
686938f5db [fix](nereids) window function with grouping sets work not well (#31475)
```sql        
select a, c, sum(sum(b)) over(partition by c order by c rows between unbounded preceding and current row) 
from test_window_table2 group by grouping sets((a),( c)) having a > 1 order by 1,2,3;
```
for this kind of case:
sum(sum(col)) over, nereids has cannot find slot problem.
the output slot of repeat and aggregate is computed wrongly.
Only collecting the trival-agg  in NormalizeRepeat can fix this problem.


Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2024-02-29 16:44:40 +08:00
a6ab6c1cb8 [fix](nereids) stats derive for "not equal“, avoid to derive zero ndv (#31566) 2024-02-29 16:44:40 +08:00
0b5b7175d6 [fix](multi-catalog) add max compute custom odps and tunnel url (#31390)
add max compute custom odps and tunnel url
2024-02-29 16:44:40 +08:00
b9a87c63f7 [chore](catalog recycle bin) Add option to ignore min erase latency for testing (#31417) 2024-02-29 16:44:40 +08:00
df4b289825 fix total task exec time is far more than actual (#31273) 2024-02-29 16:44:40 +08:00
9c4708ee74 [function](random_bytes)add random_bytes function (#31547)
SELECT random_bytes(10);

random_bytes(10) |
----------------------+
0x9b8ea00b7d1084bc5b26|
2024-02-29 16:44:39 +08:00
95b1f76664 [Feature](executor)broker load support workload group (#30866) (#31580) 2024-02-29 15:09:10 +08:00
c60fea9bdf [fix](mtmv)fix getIdToItem cause ConcurrentModificationException (#31511) 2024-02-29 12:38:03 +08:00