Commit Graph

5976 Commits

Author SHA1 Message Date
7667fe8570 [Improve)(Variant) do not allow fall back to legacy planner (#30430) 2024-01-29 19:02:46 +08:00
658c869aac [improvement](mtmv)mtmv support partition by hms table (#29989) 2024-01-29 19:02:46 +08:00
15a68924f5 Add note for workload group when upgrade Doris (#30457) 2024-01-29 19:02:45 +08:00
bfdc41d37b [fix](ccr) handle large binlog (#30435) 2024-01-28 18:25:31 +08:00
92dc395f9a [fix](nereids)should always call visitBoundFunction first when binding ElementAt function (#30469) 2024-01-28 18:25:31 +08:00
0433b8730d [Feature](profile)add shuffle send rows/bytes #30456 2024-01-28 18:25:08 +08:00
b1a9370004 [fix](glue)support access glue iceberg with credential list (#30473)
merge from #30292
2024-01-28 18:23:07 +08:00
e9218861ec fix code format 2024-01-27 10:40:27 +08:00
4f915129a9 [pipelineX](localexchange) Add local exchange before TabletFunction (#30446)
* [pipelineX](localexchange) Add local exchange before TabletFunction

* update
2024-01-27 10:29:41 +08:00
5d7543b30b [feature](ranger) Support Apache ranger for Doris (#27864)
For usage, see:
5d340ce24f/docs/zh-CN/docs/admin-manual/privilege-ldap/ranger.md

For range-doris-plugin, see:
https://github.com/morningman/ranger/tree/doris-plugin

To support ranger, there are several other modification:

1. Support `show resources like "pattern"`
2. Support `show workload group like "pattern"`
3. Support `show schemas like "pattern"`
2024-01-27 10:29:38 +08:00
2284575afa [opt](nereids)set flag to indicate if bloom filter size is calculated by ndv (#30278)
set flag to indicate if bloom filter size is calculated by ndv
2024-01-27 10:07:41 +08:00
5986d5415e [opt](Nereids) make runtime filter target support expression (#30131)
the target expression should be:
1. only one numeric slot, or
2. cast for any data type

example:
select * from T1 join T2 on abs(T1.a) = T2.a
RF T2.a->abs(T1.a)
2024-01-27 10:07:10 +08:00
e576412a56 fix iceberg table get split fail when with date type conjuct (#30162) 2024-01-27 09:13:21 +08:00
d51cbf00f4 [minor fix](use_fix_replica) handle set use_fix_replica < 0 (#30358) 2024-01-27 09:11:44 +08:00
5d2b011a8d [fix](ES catalog) throw exception when ES meta track failed (#30409) 2024-01-27 09:11:02 +08:00
04237f60e0 [feature](Nereids): eager aggreagate support mix agg function (#30400) 2024-01-27 09:11:02 +08:00
1fdb323154 [pipeline](load) p0 load failed in pipeline stream load (#30438) 2024-01-27 09:11:02 +08:00
713798d549 [feature](nereids)support mark join (#30133)
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
2024-01-27 09:09:53 +08:00
f25af15842 [Fix](Nereids) Fix lost predicate when query has a filter at the right input of the outer join (#30374)
materialized view def is as following:
>        select l_shipdate, o_orderdate, l_partkey, l_suppkey, o_orderkey  
>        from lineitem 
>        left join (select * from orders where o_orderdate = '2023-12-10' ) t2 
>        on lineitem.l_orderkey = t2.o_orderkey;
    
the query as following, should add filter `o_orderdate = '2023-12-10'` on mv when query rewrite by materialized view
>        select l_shipdate, o_orderdate, l_partkey, l_suppkey, o_orderkey 
>         from lineitem 
>         left join orders 
>        on lineitem.l_orderkey = orders.o_orderkey 
>         where o_orderdate = '2023-12-10' order by 1, 2, 3, 4, 5;
2024-01-27 09:09:02 +08:00
8543167195 [Nereids](Variant) Implement variant type and support new sub column access method (#30348)
* [Nereids](Variant) Implement variant type in Variant and support new sub column access method

The query SELECT v["a"]["b"] from simple_var WHERE cast(v["a"]["b"] as int) = 1

1. During the binding stage, the expression element_at(var, "xxx") is transformed into a SlotReference with a specified path. This conversion is tracked in the StatementContext, where the parent slot is the primary key and the paths are secondary keys. This structure, known as subColumnSlotRefMap in the StatementContext, helps to eliminate duplicates of the same slot derived from identical paths.

2. A new rule, BindSlotWithPaths, is introduced in the analysis stage. This rule is responsible for converting slots with paths into their respective slot suppliers. To ensure that slots with paths are correctly associated with the appropriate LogicalOlapScan, an additional mapping, slotToRelation, is added to the StatementContext. This mapping links the top-level slot to its corresponding relation (i.e., LogicalOlapScan). Consequently, subsequent slots with paths can determine the correct LogicalOlapScan to merge with and modify accordingly.
2024-01-27 09:09:02 +08:00
Pxl
1c449d3204 [Improvement](priv) move check priv out of analyze (#30403)
move check priv out of analyze
2024-01-27 09:09:02 +08:00
9aa3cdfa1b Change auto analyze max table width default value to 100. (#30395) 2024-01-27 09:08:29 +08:00
4bdc2cde2a [chore](Nereids) enable nereids if update from doris 1.x (#30369) 2024-01-27 09:08:29 +08:00
a954bab81a [fix](function) fix error result in time_to_sec and timediff (#30248) 2024-01-27 09:08:29 +08:00
09ae37dcc5 [pipelineX](localexchange) Adjust local exchange plan rule (#30393) 2024-01-27 09:08:29 +08:00
9857a38b63 [fix](fe) Fix BackendHbResponse serialization compatibility problem (#30441)
* Similar to https://github.com/apache/doris/pull/30337

* Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade
  compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion
  less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField`

* Introduced by https://github.com/apache/doris/pull/24314
2024-01-26 17:58:14 +08:00
2ae02efd46 [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog (#26769)
create catalog in local test/dev env like below:
create catalog iceberg PROPERTIES ('type'='iceberg','iceberg.catalog.type' = 'hadoop', 'warehouse' = '/export/workspace/warehouse');

we will get error: "Unrecognized 'warehouse' location format because name service is required."
2024-01-25 23:56:39 +08:00
7938531d1e [feature](load) enable memtable on sink node by default (#30372) 2024-01-25 21:37:33 +08:00
ce0b4fed6d [feat](log) Update RestApiExceptionHandler log level from debug to warn (#30306)
* When meeting RestApiExceptionHandler, we have insufficient information
  for finding out the call stack
2024-01-25 21:37:33 +08:00
3ffd5b8e9a [fix](Nereids) some special expression should not be constant (#30305) 2024-01-25 21:37:10 +08:00
4681958cf7 [improvement](replica) set replica drop write editlog (#30345)
* set replica drop write editlog

* update test
2024-01-25 21:37:10 +08:00
e8a20d8899 [fix](meta) remove default_cluster prefix in recover db log (#30350) 2024-01-25 21:33:51 +08:00
92d4ce31ae [improve](routine-load) optimize error msg when failed to fetch Kafka info #30298 2024-01-25 21:33:50 +08:00
0f81ecf415 [feat](Nereids): eliminate inner join by pk fk when comparing mv (#30258) 2024-01-25 21:33:50 +08:00
0a5c375068 [fix](Nereids): when predicate contains right output, don't convert outer to anti join (#30276) 2024-01-25 21:33:50 +08:00
bb021668c9 [feat](log) Add more log about cost time for dropTable (#30284)
* In some p0 test cases we have found dropTable cost more than
  5 seconds, so add more time cost log
2024-01-25 14:32:39 +08:00
f87484d6b3 [fix](Nereids) fix create array function type coercion (#30329) 2024-01-25 13:24:52 +08:00
081da1299a [fix](fe) Fix FrontendHbResponse serialization compatibility problem (#30337)
* Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade
  compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion
  less than FeMetaVersion.VERSION_121 will call `FrontendHbResponse.readField`

* Introduced by https://github.com/apache/doris/pull/24314
2024-01-25 13:24:52 +08:00
Pxl
5b462194d1 [Feature](materialized-view) support rewrite case when to if on legacy planner to make mv work (#30320)
support rewrite case when to if on legacy planner to make mv work
2024-01-25 13:24:52 +08:00
3d22f9cfc8 [feature](replica) Add drop replica safely on backend (#30303) 2024-01-25 13:24:52 +08:00
bee6ae73c7 [minor](Nereids): enable PushDownTopNDistinctThroughJoin (#30275) 2024-01-25 13:24:52 +08:00
0287651114 [fix](char) fix char column type not check lenght (#30266) 2024-01-25 13:24:09 +08:00
86d7a8be44 [improvement](statistics nereids)Nereids support select mv. (#30267) 2024-01-25 13:24:09 +08:00
63cf2777f9 [improvement](statistics)Catch load column stats exception, avoid print too much stack info to fe.out #30315 2024-01-25 13:24:09 +08:00
83ea486b15 [fix](Nereids): Except just can merge with left deep shape (#30270) 2024-01-25 13:24:09 +08:00
Pxl
7e60369ba2 [Feature](materialized-view) support create mv with count(*) (#30313)
support create mv with count(*)
2024-01-25 13:24:09 +08:00
2951f9a2c7 [fix](Nereids) relation in constraints should be multi parts (#30293) 2024-01-25 13:24:09 +08:00
7e1a986fa1 [feat](Nereids): eliminate left outer join by unique when comparing mv (#30228) 2024-01-25 13:24:09 +08:00
cd70f45ce2 [test](ut) added UT cases for show build index (#29561)
Added UT case for show build index flow
2024-01-25 13:24:09 +08:00
ca5a314765 [fix](function) make STRLEFT and STRRIGHT and SUBSTR function DEPEND_ON_ARGUMENT (#28352)
make STRLEFT and STRRIGHT function DEPEND_ON_ARGUMENT
2024-01-25 13:23:59 +08:00