Commit Graph

5947 Commits

Author SHA1 Message Date
3ffd5b8e9a [fix](Nereids) some special expression should not be constant (#30305) 2024-01-25 21:37:10 +08:00
4681958cf7 [improvement](replica) set replica drop write editlog (#30345)
* set replica drop write editlog

* update test
2024-01-25 21:37:10 +08:00
e8a20d8899 [fix](meta) remove default_cluster prefix in recover db log (#30350) 2024-01-25 21:33:51 +08:00
92d4ce31ae [improve](routine-load) optimize error msg when failed to fetch Kafka info #30298 2024-01-25 21:33:50 +08:00
0f81ecf415 [feat](Nereids): eliminate inner join by pk fk when comparing mv (#30258) 2024-01-25 21:33:50 +08:00
0a5c375068 [fix](Nereids): when predicate contains right output, don't convert outer to anti join (#30276) 2024-01-25 21:33:50 +08:00
bb021668c9 [feat](log) Add more log about cost time for dropTable (#30284)
* In some p0 test cases we have found dropTable cost more than
  5 seconds, so add more time cost log
2024-01-25 14:32:39 +08:00
f87484d6b3 [fix](Nereids) fix create array function type coercion (#30329) 2024-01-25 13:24:52 +08:00
081da1299a [fix](fe) Fix FrontendHbResponse serialization compatibility problem (#30337)
* Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade
  compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion
  less than FeMetaVersion.VERSION_121 will call `FrontendHbResponse.readField`

* Introduced by https://github.com/apache/doris/pull/24314
2024-01-25 13:24:52 +08:00
Pxl
5b462194d1 [Feature](materialized-view) support rewrite case when to if on legacy planner to make mv work (#30320)
support rewrite case when to if on legacy planner to make mv work
2024-01-25 13:24:52 +08:00
3d22f9cfc8 [feature](replica) Add drop replica safely on backend (#30303) 2024-01-25 13:24:52 +08:00
bee6ae73c7 [minor](Nereids): enable PushDownTopNDistinctThroughJoin (#30275) 2024-01-25 13:24:52 +08:00
0287651114 [fix](char) fix char column type not check lenght (#30266) 2024-01-25 13:24:09 +08:00
86d7a8be44 [improvement](statistics nereids)Nereids support select mv. (#30267) 2024-01-25 13:24:09 +08:00
63cf2777f9 [improvement](statistics)Catch load column stats exception, avoid print too much stack info to fe.out #30315 2024-01-25 13:24:09 +08:00
83ea486b15 [fix](Nereids): Except just can merge with left deep shape (#30270) 2024-01-25 13:24:09 +08:00
Pxl
7e60369ba2 [Feature](materialized-view) support create mv with count(*) (#30313)
support create mv with count(*)
2024-01-25 13:24:09 +08:00
2951f9a2c7 [fix](Nereids) relation in constraints should be multi parts (#30293) 2024-01-25 13:24:09 +08:00
7e1a986fa1 [feat](Nereids): eliminate left outer join by unique when comparing mv (#30228) 2024-01-25 13:24:09 +08:00
cd70f45ce2 [test](ut) added UT cases for show build index (#29561)
Added UT case for show build index flow
2024-01-25 13:24:09 +08:00
ca5a314765 [fix](function) make STRLEFT and STRRIGHT and SUBSTR function DEPEND_ON_ARGUMENT (#28352)
make STRLEFT and STRRIGHT function DEPEND_ON_ARGUMENT
2024-01-25 13:23:59 +08:00
88fdb2677d [fix](catalog) fix Uninitialized connection pool parameters in hmsjdbcclient (#30262) 2024-01-25 13:23:59 +08:00
2f68aac885 [Improvement](Nereids) Support to query rewrite by materialized view when join input has aggregate (#30230)
Support to query rewrite by materialized view when join input has aggregate, the aggregate should be simple
For example as following:
The materialized view def is 
>            select
>              l_linenumber,
>              count(distinct l_orderkey),
>              sum(case when l_orderkey in (1,2,3) then l_suppkey * l_linenumber else 0 end),
>              max(case when l_orderkey in (4, 5) then (l_quantity *2 + part_supp_a.qty_max) * 0.88 else 100 end),
>              avg(case when l_partkey in (2, 3, 4) then l_discount + o_totalprice + part_supp_a.qty_sum else 50 end)
>            from lineitem
>            left join orders on l_orderkey = o_orderkey
>            left join 
>              (select ps_partkey, ps_suppkey, sum(ps_availqty) qty_sum, max(ps_availqty) qty_max,
>                min(ps_availqty) qty_min,
>                avg(ps_supplycost) cost_avg
>                from partsupp
>                group by ps_partkey,ps_suppkey) part_supp_a
>              on l_partkey = part_supp_a.ps_partkey
>                and l_suppkey = part_supp_a.ps_suppkey
>            group by l_linenumber;

when query is like following, it can be rewritten by mv above
>            select
>              l_linenumber,
>              sum(case when l_orderkey in (1,2,3) then l_suppkey * l_linenumber else 0 end),
>              avg(case when l_partkey in (2, 3, 4) then l_discount + o_totalprice + part_supp_a.qty_sum else 50 end)
>            from lineitem
>            left join orders on l_orderkey = o_orderkey
>            left join 
>              (select ps_partkey, ps_suppkey, sum(ps_availqty) qty_sum, max(ps_availqty) qty_max,
>                min(ps_availqty) qty_min,
>                avg(ps_supplycost) cost_avg
>                from partsupp
>                group by ps_partkey,ps_suppkey) part_supp_a
>              on l_partkey = part_supp_a.ps_partkey
>                and l_suppkey = part_supp_a.ps_suppkey
>            group by l_linenumber;
2024-01-25 13:23:59 +08:00
f85b04c2c6 [fix](datatype) fixed decimal type implicit cast handling in BinaryPredicate (#30181) 2024-01-25 13:23:12 +08:00
c7360fd014 [feature](function) support ip function named ipv4_cidr_to_range(addr, cidr) (#29819)
* support ip function ipv4_cidr_to_range

* fix ipv4_cidr_to_range function only support ipv4 type
2024-01-24 10:02:03 +08:00
dde5ed5231 [fix](fe-memory) Fix fe schema change high memory usage (#30231)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2024-01-24 10:02:03 +08:00
2b3e7589b7 [fix](nereids)group by expr may be lost in EliminateGroupByConstant rule (#30274) 2024-01-24 10:01:14 +08:00
4af3fd2a2e [fix](Nereids) fix bug in case-when/if stats estimation (#30265) 2024-01-24 10:00:25 +08:00
72f4e7e2d1 [security] Don't print token (#30227) 2024-01-24 09:59:45 +08:00
b98acf2d90 [fix](mtmv)mtmv default open enableNereidsDML #30235 2024-01-24 09:59:45 +08:00
4cbacb5b39 [enhancement](recover) Support skipping bad tablet in select by session variable (#30241)
In some scenarios, user has a huge amount of data and only a single replica was specified when creating the table, if one of the tablet is damaged, the table will not be able to be select. If the user does not care about the integrity of the data, they can use this variable to temporarily skip the bad tablet for querying and load the remaining data into a new table.
2024-01-24 09:59:43 +08:00
1b9f1f6483 [feature](Planner): Push down TopNDistinct through Join (#30216)
Push down TopNDistinct through Outer/Cross Join
2024-01-24 09:59:13 +08:00
f4a10c3fbc [enhancement](Nereids): add builder for hyper graph (#30061) 2024-01-24 09:58:31 +08:00
9a8bcf2b1b [fix](planner) row policy rewriter generate wrong plan on join table ref (#30233) 2024-01-23 14:11:54 +08:00
2499ca6d89 [Enhancement](plan) Optimize preagg for aggregate function (#28886) 2024-01-23 13:22:14 +08:00
Pxl
1e74ad3f3b [Feature](materialized-view) support predicate apprear both on key and value mv column (#30215)
support predicate apprear both on key and value mv column
2024-01-23 13:22:14 +08:00
0e5d56fc2e [pipelineX](fix) Fix use-after-free MultiCastSourceDependency (#30199) 2024-01-23 13:22:14 +08:00
510d88f315 [fix](mtmv)return MTMV with at least one available partition #30156 2024-01-23 10:12:37 +08:00
32c5153999 [fix](routine-load) pause job when json path is invalid #30197
If jsonpaths is set wrong, routine load job will report error but running all time.For example:

CREATE ROUTINE LOAD jobName ON tableName
PROPERTIES
(
    "format" = "json",
    "max_batch_interval" = "5",
    "max_batch_rows" = "300000",
    "max_batch_size" = "209715200",
    "jsonpaths" = "[\'t\',\'a\']"
)
FROM KAFKA
(
    "kafka_broker_list" = "$IP:PORT",
    "kafka_topic" = "XXX",
    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
);
Jsonpaths ['t','a'] is invalid, but job will running all time.
2024-01-23 10:12:37 +08:00
9c742d46a2 [fix](group commit) abort txn should use label if replay wal failed (#30219) 2024-01-23 10:12:35 +08:00
9e0c518aaf [Feature](executor)Workload Group support Non-Pipeline Execution (#30164) 2024-01-23 10:11:25 +08:00
ce47354d59 [fix](Nereids) result nullable of sum distinct in scalar agg is wrong (#30221) 2024-01-23 10:09:54 +08:00
d5d0e5e611 [feature](function) support ip functions named to_ipv4[or_default, or_null](string) and to_ipv6[or_default, or_null](string) (#29838) 2024-01-23 10:09:54 +08:00
xy
45f6cba837 [fix](Nereids) Fixed a bug where the execution plan was incorrect after ddl (#30107)
should only compare column name when generate data dist info of PhysicalOlapScan

Co-authored-by: xingying01 <xingying01@corp.netease.com>
2024-01-23 10:09:54 +08:00
8061597f2a [fix](Nereids) nullable not adjust in output exprs in result sink node (#30206) 2024-01-23 10:09:54 +08:00
5c43708d92 [Fix](Job)Incorrect task query result of insert type (#30024)
- IdToTask has no persistence, so the queried task will be lost once it is restarted.

- The cancel task does not update metadata after being removed from the running task.

- tvf displays an error when some fields in the query task result are empty

- cycle scheduling job should not be STOP when task fail
2024-01-23 10:09:54 +08:00
24c0900b41 [fix](planner) should return outputTupleDesc's id instead of tupleIds if outputTupleDesc is set in Plan Node (#30150) 2024-01-23 10:09:54 +08:00
e5dea910bf [feature](bitwise function) bit_count/bit_shift_left/bit_shift_right implementation (#30046) 2024-01-23 10:09:54 +08:00
62a46876b6 [improvement](statistics) Optimize drop stats operation (#30144)
Before, drop stats operation need to call columns * followers times of isMaster() function and the same times of rpc to drop remote column stats. This pr is to reduce the rpc calls and use more efficient way to check master node instead of using isMaster()
2024-01-23 10:09:54 +08:00
3e73933857 [fix](routineload) check offset when schedule tasks (#30136) 2024-01-23 10:09:54 +08:00