Commit Graph

17549 Commits

Author SHA1 Message Date
dce6c8bd65 [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression (#30440)
materialized view definition is

>            select
>            sum(o_totalprice) as sum_total,
>            max(o_totalprice) as max_total,
>            min(o_totalprice) as min_total,
>           count(*) as count_all,
>            bitmap_union(to_bitmap(case when o_shippriority > 1 and o_orderkey IN (1, 3) then o_custkey else null end)) >cnt_1,
>            bitmap_union(to_bitmap(case when o_shippriority > 2 and o_orderkey IN (2) then o_custkey else null end)) as >cnt_2
>            from lineitem
>            left join orders on l_orderkey = o_orderkey and l_shipdate = o_orderdate;
   

the query following can be rewritten by materialized view above.
it use the aggregate fuction arithmetic calculation in the select 

>            select
>            count(distinct case when O_SHIPPRIORITY > 2 and o_orderkey IN (2) then o_custkey else null end) as cnt_2,
>            (sum(o_totalprice) + min(o_totalprice)) * count(*),
>            min(o_totalprice) + count(distinct case when O_SHIPPRIORITY > 2 and o_orderkey IN (2) then o_custkey else null >end)
>            from lineitem
>            left join orders on l_orderkey = o_orderkey and l_shipdate = o_orderdate;
2024-01-29 19:03:47 +08:00
edeec320d3 [test](nereids) add SimplifyCastRule test case (#26708) 2024-01-29 19:03:47 +08:00
081cdc6ecd [test](nereids)add more case for SimplifyRange rule (#27314) 2024-01-29 19:03:47 +08:00
e36f390134 [fix](nereids)window expression's window frame may lost in NormalizeToSlot (#30378) 2024-01-29 19:03:47 +08:00
036e17dcb0 [test](nereids)add fe ut for SimplifyArithmeticComparisonRule (#27644) 2024-01-29 19:03:47 +08:00
3b85e3de1b [fix](planner)avg function may use wrong decimal precision and scale (#30364) 2024-01-29 19:03:47 +08:00
90c0806178 [fix](query state) Print correct DML state (#30489) 2024-01-29 19:03:47 +08:00
b91a338bce [enhance](auto-partition) Constrain dynamic & auto partition use same interval unit if both enable (#30426) 2024-01-29 19:03:47 +08:00
3354ac48f7 [enhance](mtmv)add version and version time for table (#30437)
Add version to record data changes in the table

Scope of impact: 

- Transaction related operations
- drop partition
- replace partition
2024-01-29 19:03:47 +08:00
db094da081 [fix](common) If the properties in DDL is not provided, the mysql client will lost connection (#30256) 2024-01-29 19:03:47 +08:00
11f1b129c0 [optimize](invert index) avoid redundant checks for exist. (#30191) 2024-01-29 19:03:47 +08:00
779a9a1fbb [opt](planner) use string for varchar in ctas if original table is not olap (#30323) 2024-01-29 19:03:47 +08:00
930e3bb701 [feature](Nereids): double eager support mix function (#30468) 2024-01-29 19:03:47 +08:00
cc963b0f71 [Refact](inverted index) use boost regex to resolve stack overflow issues (#30477) 2024-01-29 19:02:46 +08:00
dcfccde3d1 [fix](Nereids) create table should check column name format (#30421) 2024-01-29 19:02:46 +08:00
5a13c7596a [fix](nereids)should normalize mv column's name before matching prefix keys (#27464) 2024-01-29 19:02:46 +08:00
ae38f28280 [feature](invert index) does not create an inverted index to support the match_phrase_prefix feature. (#30414) 2024-01-29 19:02:46 +08:00
7667fe8570 [Improve)(Variant) do not allow fall back to legacy planner (#30430) 2024-01-29 19:02:46 +08:00
658c869aac [improvement](mtmv)mtmv support partition by hms table (#29989) 2024-01-29 19:02:46 +08:00
a4ccf92fec [fix](thirdparty) patch brpc 1.4.0 to fix stream rpc (#30476) 2024-01-29 19:02:46 +08:00
15a68924f5 Add note for workload group when upgrade Doris (#30457) 2024-01-29 19:02:45 +08:00
bfdc41d37b [fix](ccr) handle large binlog (#30435) 2024-01-28 18:25:31 +08:00
92dc395f9a [fix](nereids)should always call visitBoundFunction first when binding ElementAt function (#30469) 2024-01-28 18:25:31 +08:00
7e19224a6c [fix](function) fix ipv4 funcs get failed error, improve an ipv6 func and exception message (#30269) 2024-01-28 18:25:31 +08:00
0433b8730d [Feature](profile)add shuffle send rows/bytes #30456 2024-01-28 18:25:08 +08:00
b1a9370004 [fix](glue)support access glue iceberg with credential list (#30473)
merge from #30292
2024-01-28 18:23:07 +08:00
f988686708 2.1.0-rc07 2024-01-27 10:55:03 +08:00
96c4fcfb20 [improve](node) refactor partition sort node to reduce memory use
pipelineX
2024-01-27 10:54:29 +08:00
e9218861ec fix code format 2024-01-27 10:40:27 +08:00
4f915129a9 [pipelineX](localexchange) Add local exchange before TabletFunction (#30446)
* [pipelineX](localexchange) Add local exchange before TabletFunction

* update
2024-01-27 10:29:41 +08:00
5d7543b30b [feature](ranger) Support Apache ranger for Doris (#27864)
For usage, see:
5d340ce24f/docs/zh-CN/docs/admin-manual/privilege-ldap/ranger.md

For range-doris-plugin, see:
https://github.com/morningman/ranger/tree/doris-plugin

To support ranger, there are several other modification:

1. Support `show resources like "pattern"`
2. Support `show workload group like "pattern"`
3. Support `show schemas like "pattern"`
2024-01-27 10:29:38 +08:00
2284575afa [opt](nereids)set flag to indicate if bloom filter size is calculated by ndv (#30278)
set flag to indicate if bloom filter size is calculated by ndv
2024-01-27 10:07:41 +08:00
5986d5415e [opt](Nereids) make runtime filter target support expression (#30131)
the target expression should be:
1. only one numeric slot, or
2. cast for any data type

example:
select * from T1 join T2 on abs(T1.a) = T2.a
RF T2.a->abs(T1.a)
2024-01-27 10:07:10 +08:00
823c469c5c [fix](rowsetreader) determine merge iterator considering segment num (#29269) 2024-01-27 09:13:21 +08:00
e576412a56 fix iceberg table get split fail when with date type conjuct (#30162) 2024-01-27 09:13:21 +08:00
bedad15f03 [enhancement](scanner) add a lower bound for bytes in scanner queue (#29624) 2024-01-27 09:13:21 +08:00
f571ffe57f (fix)[group commit] Row count is incorrect when enable pipeline load (#30447) 2024-01-27 09:13:21 +08:00
d191809372 [fix](pipeline) Fix non-prepared execute of UnionOperator (#30355) 2024-01-27 09:11:44 +08:00
d51cbf00f4 [minor fix](use_fix_replica) handle set use_fix_replica < 0 (#30358) 2024-01-27 09:11:44 +08:00
46cadc9856 [minor](Prefetch) log slow prefetch io operation #30415 2024-01-27 09:11:02 +08:00
5d2b011a8d [fix](ES catalog) throw exception when ES meta track failed (#30409) 2024-01-27 09:11:02 +08:00
04237f60e0 [feature](Nereids): eager aggreagate support mix agg function (#30400) 2024-01-27 09:11:02 +08:00
cea9cf07a4 [docs](fix) Fix 2.0.4 Release note for master #30434 2024-01-27 09:11:02 +08:00
1fdb323154 [pipeline](load) p0 load failed in pipeline stream load (#30438) 2024-01-27 09:11:02 +08:00
f1a135619c [fix](fe) Fix BackendHbResponse serialization compatibility problem (#30433)
* Similar to https://github.com/apache/doris/pull/30337

* Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade
  compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion
  less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField`

* Introduced by https://github.com/apache/doris/pull/24314
2024-01-27 09:10:50 +08:00
22322a3864 [refactor](join) remove unused RowRefListType from join (#30442) 2024-01-27 09:10:41 +08:00
904182685b [debug](move-memtable) enable brpc debug log in regression pipelines (#30389) 2024-01-27 09:10:41 +08:00
fe082e374f [doc](fix) K8s doc modify configmap code (#30420) 2024-01-27 09:10:41 +08:00
713798d549 [feature](nereids)support mark join (#30133)
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
2024-01-27 09:09:53 +08:00
f25af15842 [Fix](Nereids) Fix lost predicate when query has a filter at the right input of the outer join (#30374)
materialized view def is as following:
>        select l_shipdate, o_orderdate, l_partkey, l_suppkey, o_orderkey  
>        from lineitem 
>        left join (select * from orders where o_orderdate = '2023-12-10' ) t2 
>        on lineitem.l_orderkey = t2.o_orderkey;
    
the query as following, should add filter `o_orderdate = '2023-12-10'` on mv when query rewrite by materialized view
>        select l_shipdate, o_orderdate, l_partkey, l_suppkey, o_orderkey 
>         from lineitem 
>         left join orders 
>        on lineitem.l_orderkey = orders.o_orderkey 
>         where o_orderdate = '2023-12-10' order by 1, 2, 3, 4, 5;
2024-01-27 09:09:02 +08:00