mv in select materialized_view should disable show table,
because Nereids planner can output the string such as
slot#[0] in toSql() of SlotRef. Note this is only a
temporary solution, will use an expression translator later
select rank() over (partition by A, B) as r, sum(x) over(A, C) as s from T;
A is a common partition key for all windowExpressions, that is A is intersection of {A,B} and {A, C}
we could push filter A=1 through this window, since A is a common Partition key:
select * from (select a, row_number() over (partition by a) from win) T where a=1;
origin plan:
----filter((T.a = 1))
----------PhysicalWindow
------------PhysicalQuickSort
--------------PhysicalProject
------------------PhysicalOlapScan[win]
transformed to
----PhysicalWindow
------PhysicalQuickSort
--------PhysicalProject
----------filter((T.a = 1))
------------PhysicalOlapScan[win]
But C=1 can not be pushed through window.
Issue Number: close #xxx
ShowTableStmtTest.testNoDb and DropDbStmtTest.testNoPriv are unstable cases，error msg is:
java.lang.Exception: Unexpected exception, expected<org.apache.doris.common.AnalysisException> but was<mockit.internal.expectations.invocation.MissingInvocation>
we can not know what is missing ,and this issue cannot be reproduced locally,so add some log
When be load reblancer choose candidate tablets, it will try moving tablets from high load backends to low backend backends. If the higher HIGH BE has no available slot num, it should try next HIGH BE.
Sometimes FE replica's version is unreliable. FE's replica may bigger than BE's real version. Need check if BE missing version (last failed version > 0).
void throw analysis ex for unsupported type to make ext table goes nereids.
this will improve the nereids' availability for external table if unsupported type is in the basic table schema but not referenced in the real sql.
tested in external table env.
Consider the following case:
select pu.pk_ct_pu as id
from fms_rd_nc65_zb.NC65P.CT_PU pu
left join fms_rd_nc65_zb.NC65P.PUB_WF_INSTANCE pwi
on pu.pk_ct_pu = pwi.billid
and pu.vtrantypecode=pwi.billtype
left join fms_rd_nc65_zb.NC65P.SM_USER su
on pu.creator = su.cuserid
where pu.pk_ct_pu='1001A110000000K8XPVN';
PUB_WF_INSTANCE table has a BLOB type column and currently it will throw analysis exception and fallback to old optimizer, although this column is not referenced in the real sql. The old optimizer doesn't have the outer join -> inner join rule and the "pu.pk_ct_pu='1001A110000000K8XPVN'; " is not pushed down and the performance will drop down. After the pr, we add the unsupported type instead of throw exception directly, it will decide the unused case and can continue goes nereids and use all advanced optimization for the sql.
1. The catalog is connected to an old version of Doris, and an error is reported when using the datetime field type on the Doris surface .
2. error message [fe]:
Caused by: java.lang.NumberFormatException: For input string: "DATETIM"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_291]
at java.lang.Integer.parseInt(Integer.java:580) ~[?:1.8.0_291]
at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0_291]
at org.apache.doris.datasource.jdbc.client.JdbcMySQLClient.dorisTypeToDoris(JdbcMySQLClient.java:401) ~[doris-fe.jar:1.2-
SNAPSHOT]
In the original logic, the `Export` statement generates `Selectstmt` for execution. But there is no way to make the `SelectStmt` use the new optimizer.
Now, we change the `Export` statement to generate the `outfile SQL`, and then use the new optimizer to parse the SQL so that outfile can use the new optimizer.
Before, Statistics Cleaner only handles olap db and table.
External db and tables would be removed without verification. So that external stats could stored no more than 2 days, which is the interval of Stats cleaner thread.
This pr is to add verification for external db and tables.
Nereids make TVF number as Gather distribution if backend num = 1.
But coordinator could not process gather fragment with scan node.
In the long run, we need to get coordinators to support this scenario.
But it is need a lot of refactor. So, we just forbid Gather distribution
for ScanNode now.
for example:
```sql
select avg(c3) from (select c2 from t2) v;
```
the error msg before this PR
```
Invalid call to c3.getDataType() on unbound object
```
the error msg after this PR
```
Unknown column 'c3' in 'table list' in AGGREGATE clause
```
when output meet order by not meet distribution. we use a trick way to
do enforce by set current output to any. but when we do enforce later,
we still use the old output. So when we do choose best plan, we could
not find the older output's plan, since we have replace it by any.
For example:
```
lowest Plan(cost, properties, plan, childrenRequires)
18.0 ANY
id:138#4 cost=0 [0/0/0/] estRows=4 children=[@0 ] (plan=PhysicalWindow[139]@4 ( windowFrameGroup=(Funcs=[row_number() WindowSpec(PARTITION BY b#1, a#0 ROWS BETWEEN UNBOUNDED_PRECEDING AND CURRENT_ROW) AS `r1`#2], PartitionKeys=[b#1, a#0], OrderKeys=[], WindowFrame=WindowFrame(ROWS, UNBOUNDED_PRECEDING, CURRENT_ROW)), requiredProperties=[DistributionSpecHash ( orderedShuffledColumns=[1, 0], shuffleType=REQUIRE, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1], [0]], exprIdToEquivalenceSet={0=1, 1=0} ) Order: ([b#1 asc, a#0 asc])], stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[0], shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297, partitionIds=[3547295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0} ) Order: ([b#1 asc, a#0 asc])]
32.01171875 DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=REQUIRE, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ) Order: ([b#1 asc])
id:161#4 cost=14 [4/4/4/] estRows=4 children=[@4 ] (plan=PhysicalQuickSort[162]@4 ( orderKeys=[b#1 asc], phase=LOCAL_SORT, stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[0], shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297, partitionIds=[3547295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0} ) Order: ([b#1 asc, a#0 asc])]
32.01171875 DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ) Order: ([b#1 asc])
id:161#4 cost=14 [4/4/4/] estRows=4 children=[@4 ] (plan=PhysicalQuickSort[162]@4 ( orderKeys=[b#1 asc], phase=LOCAL_SORT, stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ) Order: ([])]
18.01171875 DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ) Order: ([])
id:157#4 cost=0 [0/0/0/] estRows=4 children=[@4 ] (plan=PhysicalDistribute[158]@4 ( distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ), stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[0], shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297, partitionIds=[3547295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0} ) Order: ([b#1 asc, a#0 asc])]
```
the last one require a natural shuffle type property from this group.
but this property already been removed when we do
enforceDistributionButMeetSort. So, such exception will be thrown
```
Caused by: org.apache.doris.nereids.exceptions.AnalysisException: Failed to choose best plan
at org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:340) ~[classes/:?]
at org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:323) ~[classes/:?]
... 18 more
Caused by: org.apache.doris.nereids.exceptions.AnalysisException: lowestCostPlans with physicalProperties(DistributionSpecHash ( orderedShuffledColumns=[0], shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297, partitionIds=[35
47295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0} ) Order: ([b#1 asc, a#0 asc])) doesn't exist in root group
at org.apache.doris.nereids.NereidsPlanner.lambda$chooseBestPlan$1(NereidsPlanner.java:318) ~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:408) ~[?:?]
at org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:317) ~[classes/:?]
at org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:323) ~[classes/:?]
... 18 more
```
This function requires one arguments just as ARRAY_AGG(col) and col means the column whose values you want to aggregate.
This function Aggregates the values including NULL in a column into an array and returns a value of the ARRAY data type.
- support insert overwrite for Nereids
- Solving the problem of table ID changing（When no partition is specified，automatically specify all partitions for table ）
todo:
- Under the current processing logic, if the master restarts midway, it will generate dirty data (temp partition)
- for old planner ,insert overwrite table ,table id will change
or expression context can't propagation cross or expression.
for example:
```
select (a = 1 or a = 2 or a = 3) + (a = 4 or a = 5 or a = 6)
= select a in [1, 2, 3] + a in [4,5,6]
!= select a in [1, 2, 3] + a in [1, 2, 3, 4, 5, 6]
```
create table t1(c1 int not null, c2 int not null) distributed by hash(c1) buckets 3 PROPERTIES ("replication_allocation" = "tag.location.default: 1");
create table t2(c1 int not null, c2 int not null) distributed by hash(c1) buckets 3 PROPERTIES ("replication_allocation" = "tag.location.default: 1");
select t2.* from t1 left outer t2 where t2.c1 is null
==>
t1 right anti t2
