Commit Graph

7715 Commits

Author SHA1 Message Date
54709ecf3b [improvement](plsql) Select statement supports insert into variables #31574 2024-02-29 19:51:18 +08:00
92e3b31f50 [feature](invert index) match_phrase_edge feature added (#31142) 2024-02-29 19:51:18 +08:00
de28d7cd2d [fix](Nereids): AssertNumRowsElement shouldn't be expression (#31581) 2024-02-29 19:51:07 +08:00
720b6e3d86 [fix](nereids) forbid create table with illegal auto partition expr (#31498) (#31604) 2024-02-29 18:08:37 +08:00
01e6798049 [fix](Neredis) Fix npe when plan node doesn't have expressions in materialized view (#31579) 2024-02-29 16:44:40 +08:00
9988d6f9fb [Fix](executor)Fix insert select forward carry workload_group #31578 2024-02-29 16:44:40 +08:00
686938f5db [fix](nereids) window function with grouping sets work not well (#31475)
```sql        
select a, c, sum(sum(b)) over(partition by c order by c rows between unbounded preceding and current row) 
from test_window_table2 group by grouping sets((a),( c)) having a > 1 order by 1,2,3;
```
for this kind of case:
sum(sum(col)) over, nereids has cannot find slot problem.
the output slot of repeat and aggregate is computed wrongly.
Only collecting the trival-agg  in NormalizeRepeat can fix this problem.


Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2024-02-29 16:44:40 +08:00
a6ab6c1cb8 [fix](nereids) stats derive for "not equal“, avoid to derive zero ndv (#31566) 2024-02-29 16:44:40 +08:00
0b5b7175d6 [fix](multi-catalog) add max compute custom odps and tunnel url (#31390)
add max compute custom odps and tunnel url
2024-02-29 16:44:40 +08:00
b9a87c63f7 [chore](catalog recycle bin) Add option to ignore min erase latency for testing (#31417) 2024-02-29 16:44:40 +08:00
450422556a [fix](paimon) auto deplay paimon oss/s3 jar file (#31568)
No need to deploy paimon oss/s3 jar files manually.
Include them in preload-extensions-jar-with-dependencies.jar
2024-02-29 16:44:40 +08:00
df4b289825 fix total task exec time is far more than actual (#31273) 2024-02-29 16:44:40 +08:00
9c4708ee74 [function](random_bytes)add random_bytes function (#31547)
SELECT random_bytes(10);

random_bytes(10) |
----------------------+
0x9b8ea00b7d1084bc5b26|
2024-02-29 16:44:39 +08:00
95b1f76664 [Feature](executor)broker load support workload group (#30866) (#31580) 2024-02-29 15:09:10 +08:00
c60fea9bdf [fix](mtmv)fix getIdToItem cause ConcurrentModificationException (#31511) 2024-02-29 12:38:03 +08:00
6ef3455786 [fix](statistics)Fix hms external table get row count bug while analyze (#31557)
asdasd
2024-02-29 12:38:03 +08:00
17359d59a3 [fix](Nereids) reorder join generate plan is not stable (#31539) 2024-02-29 12:38:03 +08:00
Pxl
6af6997f1d [Improvement](materialized-view) add approx_count_distinct/ndv to FunctionAlias rule (#31535)
add approx_count_distinct/ndv to FunctionAlias rule
2024-02-29 12:38:03 +08:00
d75204e947 [fix](Hive-Catalog) fix NPE when using jdbc to access Hive metadata (#31559) 2024-02-29 12:38:03 +08:00
Pxl
413d733255 [Bug](materialized-view) fix npe on create mv with star (#31554)
fix npe on create mv with star
2024-02-29 12:38:03 +08:00
4a5283b466 [fix](nereids)need add mark join slot to upper project in PullUpProjectUnderApply rule (#31408) 2024-02-29 12:38:03 +08:00
3c37fb085c [refactor](jdbc catalog) split jdbc executor for different data sources (step-1) (#31406) 2024-02-29 12:38:03 +08:00
9243b3eeee [fix](multi-catalog) add config to disable external DDL (#31528)
from #31453
2024-02-29 08:42:35 +08:00
0fcdab468d [nereids] config global partition topn (#31476)
* [nereids] config global partition topn

* [nereids] config global partition topn

---------

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
2024-02-29 08:42:35 +08:00
153c775b37 [fix](Nereids) Make the case sensitivity of the result labels compatible with MySQL (#31510)
SQL: SELECT iD FROM t1
before the label was: id
after this PR the label will be: iD
2024-02-29 08:42:35 +08:00
4de25ede85 [fix](Nereids): other cond should be kept for each anti join when expanding anti join such as (#31521) 2024-02-29 08:42:35 +08:00
8633a0c0cc [Opt](exec) enable top opt in string type (#31489)
enable top opt in string type
2024-02-29 08:42:35 +08:00
3ca412efe3 Return UNKNOWN column stats if ndv is 0. (#31439) 2024-02-29 08:42:35 +08:00
e8a21b529e [Fix](fe-core) Fix The EliminateSortUnderSubquery will not affect the EliminateOrderByConstant rule (#31402) (#31403) 2024-02-28 17:52:11 +08:00
Pxl
6737fdea64 [Chore](agg-state) adjust AggStateType constructor check input (#31401)
adjust AggStateType constructor check input
2024-02-28 17:52:11 +08:00
d88caca44a [fix](Nereids) push down topn distinct through join by mistake (#31396)
should not push down topn distinct through join when the output
columns of the corresponding child of join is more than
aggregate distinct columns.

for example for LEFT_OUTER_JOIN:

left child of join's output is: c1, c2, c3.
distinct columns is: c1, c2
topn: limit 2

if we push down topn distinct, we could get result of join like this:

```
c1    c2    c3, ...
1     2     1
1     2     2
```

and the final result we get is:

```
c1    c2
1     2
```

this is wrong, because we need 2 lines, but only return 1.
2024-02-28 17:51:32 +08:00
79dd4e24ff fix compile 2024-02-28 17:38:47 +08:00
7a42d3b52c [fix](fe ut) fix TabletRepairAndBalanceTest (#31397) 2024-02-28 13:08:41 +08:00
883d022f84 [fix](paimon) fix hadoop.username does not take effect in paimon catalog (#31478) 2024-02-28 13:08:41 +08:00
d0a8a30998 [improvement](iceberg/paimon)add show table stats (#31473) 2024-02-28 13:08:36 +08:00
5bbe9f7b40 Fix replay binlog gc when not found db binlog (#31463)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2024-02-28 13:07:47 +08:00
5824f8a4bc [Feat](nereids) support multi-leading (#30379)
support multi-leading in each query block
2024-02-28 13:07:47 +08:00
ab00c7012b [fix] Fix the incorrect class name in the getLogger method call of MysqlTable. (#31465) 2024-02-28 13:07:47 +08:00
Pxl
3acfda413b [Chore](function) remove unused check on count function (#31400) 2024-02-28 13:07:47 +08:00
a371a10603 [fix](Nereids) let time type coercion same with legacy planner (#31472) 2024-02-28 13:07:47 +08:00
c0754583cb [opt](plsql) Fix procedure key compatibility (#31445)
use dbId replace dbName, because dbName may be renamed by Alter.
procedure key add package name (only reserved, currently no plans to support package)
Optimize procedure create and exception
2024-02-28 13:07:47 +08:00
9ffcf48cce [enhancement](Nereids) Support show process time and process steps by explain statement (#31339)
## Proposed changes

1. show process time when execute `explain plan xxx` by nereids
2. add `explain xxx plan process select ...` statement to show the process of the plan, not support show memo shape (physical plan) currently

example:
show process time:
```
mysql> explain plan select * from tt;
+---------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                               |
+---------------------------------------------------------------------------------------------------------------+
| ========== PARSED PLAN (time: 3ms) ==========                                                                 |
| UnboundResultSink[3] (  )                                                                                     |
| +--LogicalProject[2] ( distinct=false, projects=[*], excepts=[] )                                             |
|    +--LogicalCheckPolicy (  )                                                                                 |
|       +--UnboundRelation ( id=RelationId#0, nameParts=tt )                                                    |
|                                                                                                               |
| ========== ANALYZED PLAN (time: 6ms) ==========                                                               |
| LogicalResultSink[11] ( outputExprs=[id#0, name#1] )                                                          |
| +--LogicalProject[9] ( distinct=false, projects=[id#0, name#1], excepts=[] )                                  |
|    +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON ) |
|                                                                                                               |
| ========== REWRITTEN PLAN (time: 0ms)==========                                                               |
| LogicalResultSink[11] ( outputExprs=[id#0, name#1] )                                                          |
| +--LogicalProject[9] ( distinct=false, projects=[id#0, name#1], excepts=[] )                                  |
|    +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON ) |
|                                                                                                               |
| ========== OPTIMIZED PLAN (time: 2ms) ==========                                                              |
| PhysicalResultSink[56] ( outputExprs=[id#0, name#1] )                                                         |
| +--PhysicalDistribute[53]@1 ( stats=2, distributionSpec=DistributionSpecGather )                              |
|    +--PhysicalProject[50]@1 ( stats=2, projects=[id#0, name#1] )                                              |
|       +--PhysicalOlapScan[tt]@0 ( stats=2 )                                                                   |
+---------------------------------------------------------------------------------------------------------------+
21 rows in set (0.01 sec)
```

explain plan process:
```
mysql> explain plan process select * from tt\G
*************************** 1. row ***************************
  Rule: BINDING_RELATION
Before: UnboundResultSink[8] (  )
+--LogicalProject[7] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalCheckPolicy (  )
      +--UnboundRelation ( id=RelationId#0, nameParts=tt )
 After: UnboundResultSink[11] (  )
+--LogicalProject[10] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalCheckPolicy (  )
      +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 2. row ***************************
  Rule: CHECK_ROW_POLICY
Before: UnboundResultSink[15] (  )
+--LogicalProject[14] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalCheckPolicy (  )
      +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: UnboundResultSink[17] (  )
+--LogicalProject[16] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 3. row ***************************
  Rule: BINDING_PROJECT_SLOT
Before: UnboundResultSink[22] (  )
+--LogicalProject[21] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: UnboundResultSink[23] (  )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 4. row ***************************
  Rule: BINDING_RESULT_SINK
Before: UnboundResultSink[26] (  )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[25] ( outputExprs=[id#0, name#1] )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 5. row ***************************
  Rule: ELIMINATE_UNNECESSARY_PROJECT
Before: LogicalResultSink[25] ( outputExprs=[id#0, name#1] )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[27] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 6. row ***************************
  Rule: PRUNE_EMPTY_PARTITION
Before: LogicalResultSink[29] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[30] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 7. row ***************************
  Rule: MATERIALIZED_INDEX_SCAN
Before: LogicalResultSink[36] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[37] ( outputExprs=[id#0, name#1] )
+--LogicalProject[35] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalProject[34] ( distinct=false, projects=[id#0, name#1], excepts=[] )
      +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
*************************** 8. row ***************************
  Rule: MERGE_PROJECTS
Before: LogicalResultSink[40] ( outputExprs=[id#0, name#1] )
+--LogicalProject[39] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalProject[34] ( distinct=false, projects=[id#0, name#1], excepts=[] )
      +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[41] ( outputExprs=[id#0, name#1] )
+--LogicalProject[38] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
*************************** 9. row ***************************
  Rule: ELIMINATE_UNNECESSARY_PROJECT
Before: LogicalResultSink[42] ( outputExprs=[id#0, name#1] )
+--LogicalProject[38] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[43] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
*************************** 10. row ***************************
  Rule: REWRITE_CTE_CHILDREN
Before: LogicalResultSink[25] ( outputExprs=[id#0, name#1] )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[43] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
10 rows in set (0.00 sec)
```
2024-02-28 13:07:25 +08:00
47d112e4e6 [fix](MySQL) implement SHOW CHARSET statement. (#31389) 2024-02-28 13:07:23 +08:00
7b3377d474 [fix](Nereids) let with methods of plans use correct logical properties (#31447) 2024-02-28 13:05:57 +08:00
37590f1778 Make sure external table fetched dbId before call getRowCount. (#31379) 2024-02-28 13:05:57 +08:00
eb0416032b [feature](multi-catalog)support hms catalog create and drop table/db (#30198) (#31499)
1. rename old create/drop table to add/removeMemoryTable
2. add new create/drop table/db method
3. support hms catalog create/drop table/db

(cherry picked from commit b2e869c7414c68186de8d43b324ae736d7cc3463)
2024-02-28 09:33:54 +08:00
39a8db27f2 [fix](mtmv)TVF Query JOB Concurrent Reading and Writing Causes Exception #31422 2024-02-27 16:06:26 +08:00
6b4a756837 [Fix] Only datetime and datetimev2 types can use current_timestamp as column default value (#31395)
for this kind of sql:

create table test_default10(
  a int, 
  b varchar(100) default current_timestamp
)
distributed by hash(a)
properties('replication_num'="1");

add check:
 Types other than DATETIME and DATETIMEV2 cannot use current_timestamp as the default value
2024-02-27 16:06:26 +08:00
378ced72db [chore](Nereids) more reasonable parse select list only query (#31346) 2024-02-27 16:06:26 +08:00
481d94c3fc [feature](nereids) deal the slots that appear both in agg func and grouping sets (#31318)
this PR support slot appearing both in agg func and grouping sets.
sql like below:
select sum(a) from t group by grouping sets ((a)); 

Before this PR, Nereids throw exception like below:
col_int_undef_signed cannot both in select list and aggregate functions when using GROUPING SETS/CUBE/ROLLUP, please use union instead.

This PR removes the restriction and supports this situation.
2024-02-27 10:12:33 +08:00