Commit Graph

18429 Commits

Author SHA1 Message Date
5e80a718fd [type](docs)add example for sql_dialect #31536 2024-02-29 12:38:03 +08:00
3c37fb085c [refactor](jdbc catalog) split jdbc executor for different data sources (step-1) (#31406) 2024-02-29 12:38:03 +08:00
c956cac1a7 [enhancement](docs) Clarify JAR file locations for JDBC catalog (#31542) 2024-02-29 12:38:03 +08:00
Pxl
d36ad56dce [Opt](Exec) Support runtime update topn filter (#31250) 2024-02-29 12:38:03 +08:00
770cdabda3 [fix](group_commit) GroupCommitBlockSink shoud not use load_block_queue when creating load task fail (#31416) 2024-02-29 12:38:02 +08:00
60de835d48 [fix](meta-tool) Fix compile meta tool (#31457) 2024-02-29 12:38:02 +08:00
8c570c217e [fix](load) Fix using uint32 for tablet id overflow (#31532) 2024-02-29 12:38:02 +08:00
32033d08c6 Fix hive p2 cases. (#31541) 2024-02-29 12:37:38 +08:00
afc105647a [fix](regression) fix ssb shape case rf type (#31529) 2024-02-29 12:37:38 +08:00
b177b26d39 [branch-2.1](tracing) Pick pipeline tracing and relative bugfix (#31367)
* [Feature](pipeline) Trace pipeline scheduling (part I) (#31027)

* [fix](compile) Fix performance compile fail #31305

* [fix](compile) Fix macOS compilation issues for PURE macro and CPU core identification (#31357)

* [fix](compile) Correct PURE macro definition to fix compilation on macOS

* 2

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-02-29 08:42:35 +08:00
9243b3eeee [fix](multi-catalog) add config to disable external DDL (#31528)
from #31453
2024-02-29 08:42:35 +08:00
0fcdab468d [nereids] config global partition topn (#31476)
* [nereids] config global partition topn

* [nereids] config global partition topn

---------

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
2024-02-29 08:42:35 +08:00
153c775b37 [fix](Nereids) Make the case sensitivity of the result labels compatible with MySQL (#31510)
SQL: SELECT iD FROM t1
before the label was: id
after this PR the label will be: iD
2024-02-29 08:42:35 +08:00
ddb37d7371 Fix analyze mv from follower case bug. (#31523) 2024-02-29 08:42:35 +08:00
4de25ede85 [fix](Nereids): other cond should be kept for each anti join when expanding anti join such as (#31521) 2024-02-29 08:42:35 +08:00
ac38356058 [fix](test) fix error messages in fault injection tests (#31515) 2024-02-29 08:42:35 +08:00
3dd860606b [enhance](git)Ignore fe/gen (#31512) 2024-02-29 08:42:35 +08:00
06c80d8708 [Fix](regression) add disable_auto_compaction for show data (#31519) 2024-02-29 08:42:35 +08:00
8633a0c0cc [Opt](exec) enable top opt in string type (#31489)
enable top opt in string type
2024-02-29 08:42:35 +08:00
3ca412efe3 Return UNKNOWN column stats if ndv is 0. (#31439) 2024-02-29 08:42:35 +08:00
e8a21b529e [Fix](fe-core) Fix The EliminateSortUnderSubquery will not affect the EliminateOrderByConstant rule (#31402) (#31403) 2024-02-28 17:52:11 +08:00
Pxl
6737fdea64 [Chore](agg-state) adjust AggStateType constructor check input (#31401)
adjust AggStateType constructor check input
2024-02-28 17:52:11 +08:00
586217bf73 [Improve](Variant) support prune segment for quering variant (#31310) 2024-02-28 17:52:11 +08:00
fa020b6696 [Improvement](regression-test) generate user token by configured user and password rather than hard cord (#31345)
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
2024-02-28 17:51:32 +08:00
d88caca44a [fix](Nereids) push down topn distinct through join by mistake (#31396)
should not push down topn distinct through join when the output
columns of the corresponding child of join is more than
aggregate distinct columns.

for example for LEFT_OUTER_JOIN:

left child of join's output is: c1, c2, c3.
distinct columns is: c1, c2
topn: limit 2

if we push down topn distinct, we could get result of join like this:

```
c1    c2    c3, ...
1     2     1
1     2     2
```

and the final result we get is:

```
c1    c2
1     2
```

this is wrong, because we need 2 lines, but only return 1.
2024-02-28 17:51:32 +08:00
79dd4e24ff fix compile 2024-02-28 17:38:47 +08:00
7a42d3b52c [fix](fe ut) fix TabletRepairAndBalanceTest (#31397) 2024-02-28 13:08:41 +08:00
e3f03e10b2 [cases](regression) Add data reliability check meta cases (#31432) 2024-02-28 13:08:41 +08:00
883d022f84 [fix](paimon) fix hadoop.username does not take effect in paimon catalog (#31478) 2024-02-28 13:08:41 +08:00
f18c853495 [enhance](S3) Init default retry strategy for aws s3 sdk (#31329) 2024-02-28 13:08:36 +08:00
f1e14722d8 [docs](docs) Update 2.0.5 Release Note of EN & CN Version (#31506) 2024-02-28 13:08:36 +08:00
d0a8a30998 [improvement](iceberg/paimon)add show table stats (#31473) 2024-02-28 13:08:36 +08:00
5bbe9f7b40 Fix replay binlog gc when not found db binlog (#31463)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2024-02-28 13:07:47 +08:00
5824f8a4bc [Feat](nereids) support multi-leading (#30379)
support multi-leading in each query block
2024-02-28 13:07:47 +08:00
c5e6e3d45f [fix](docs)Correcting spelling errors,add isolated file to siderbars. (#31409)
Correcting spelling errors,add isolated file to siderbars
2024-02-28 13:07:47 +08:00
e86cc7e8e8 [chore](log) reduce a lot inject debug point log #31474 2024-02-28 13:07:47 +08:00
ab00c7012b [fix] Fix the incorrect class name in the getLogger method call of MysqlTable. (#31465) 2024-02-28 13:07:47 +08:00
Pxl
3acfda413b [Chore](function) remove unused check on count function (#31400) 2024-02-28 13:07:47 +08:00
7f566f9365 Reset report_workload_runtime_status to optional (#31479) 2024-02-28 13:07:47 +08:00
04de7aed40 [fix](case) fix testcase test_decimal256_index #31486 2024-02-28 13:07:47 +08:00
a371a10603 [fix](Nereids) let time type coercion same with legacy planner (#31472) 2024-02-28 13:07:47 +08:00
c0754583cb [opt](plsql) Fix procedure key compatibility (#31445)
use dbId replace dbName, because dbName may be renamed by Alter.
procedure key add package name (only reserved, currently no plans to support package)
Optimize procedure create and exception
2024-02-28 13:07:47 +08:00
9ffcf48cce [enhancement](Nereids) Support show process time and process steps by explain statement (#31339)
## Proposed changes

1. show process time when execute `explain plan xxx` by nereids
2. add `explain xxx plan process select ...` statement to show the process of the plan, not support show memo shape (physical plan) currently

example:
show process time:
```
mysql> explain plan select * from tt;
+---------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                               |
+---------------------------------------------------------------------------------------------------------------+
| ========== PARSED PLAN (time: 3ms) ==========                                                                 |
| UnboundResultSink[3] (  )                                                                                     |
| +--LogicalProject[2] ( distinct=false, projects=[*], excepts=[] )                                             |
|    +--LogicalCheckPolicy (  )                                                                                 |
|       +--UnboundRelation ( id=RelationId#0, nameParts=tt )                                                    |
|                                                                                                               |
| ========== ANALYZED PLAN (time: 6ms) ==========                                                               |
| LogicalResultSink[11] ( outputExprs=[id#0, name#1] )                                                          |
| +--LogicalProject[9] ( distinct=false, projects=[id#0, name#1], excepts=[] )                                  |
|    +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON ) |
|                                                                                                               |
| ========== REWRITTEN PLAN (time: 0ms)==========                                                               |
| LogicalResultSink[11] ( outputExprs=[id#0, name#1] )                                                          |
| +--LogicalProject[9] ( distinct=false, projects=[id#0, name#1], excepts=[] )                                  |
|    +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON ) |
|                                                                                                               |
| ========== OPTIMIZED PLAN (time: 2ms) ==========                                                              |
| PhysicalResultSink[56] ( outputExprs=[id#0, name#1] )                                                         |
| +--PhysicalDistribute[53]@1 ( stats=2, distributionSpec=DistributionSpecGather )                              |
|    +--PhysicalProject[50]@1 ( stats=2, projects=[id#0, name#1] )                                              |
|       +--PhysicalOlapScan[tt]@0 ( stats=2 )                                                                   |
+---------------------------------------------------------------------------------------------------------------+
21 rows in set (0.01 sec)
```

explain plan process:
```
mysql> explain plan process select * from tt\G
*************************** 1. row ***************************
  Rule: BINDING_RELATION
Before: UnboundResultSink[8] (  )
+--LogicalProject[7] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalCheckPolicy (  )
      +--UnboundRelation ( id=RelationId#0, nameParts=tt )
 After: UnboundResultSink[11] (  )
+--LogicalProject[10] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalCheckPolicy (  )
      +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 2. row ***************************
  Rule: CHECK_ROW_POLICY
Before: UnboundResultSink[15] (  )
+--LogicalProject[14] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalCheckPolicy (  )
      +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: UnboundResultSink[17] (  )
+--LogicalProject[16] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 3. row ***************************
  Rule: BINDING_PROJECT_SLOT
Before: UnboundResultSink[22] (  )
+--LogicalProject[21] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: UnboundResultSink[23] (  )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 4. row ***************************
  Rule: BINDING_RESULT_SINK
Before: UnboundResultSink[26] (  )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[25] ( outputExprs=[id#0, name#1] )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 5. row ***************************
  Rule: ELIMINATE_UNNECESSARY_PROJECT
Before: LogicalResultSink[25] ( outputExprs=[id#0, name#1] )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[27] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 6. row ***************************
  Rule: PRUNE_EMPTY_PARTITION
Before: LogicalResultSink[29] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[30] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 7. row ***************************
  Rule: MATERIALIZED_INDEX_SCAN
Before: LogicalResultSink[36] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[37] ( outputExprs=[id#0, name#1] )
+--LogicalProject[35] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalProject[34] ( distinct=false, projects=[id#0, name#1], excepts=[] )
      +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
*************************** 8. row ***************************
  Rule: MERGE_PROJECTS
Before: LogicalResultSink[40] ( outputExprs=[id#0, name#1] )
+--LogicalProject[39] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalProject[34] ( distinct=false, projects=[id#0, name#1], excepts=[] )
      +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[41] ( outputExprs=[id#0, name#1] )
+--LogicalProject[38] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
*************************** 9. row ***************************
  Rule: ELIMINATE_UNNECESSARY_PROJECT
Before: LogicalResultSink[42] ( outputExprs=[id#0, name#1] )
+--LogicalProject[38] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[43] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
*************************** 10. row ***************************
  Rule: REWRITE_CTE_CHILDREN
Before: LogicalResultSink[25] ( outputExprs=[id#0, name#1] )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[43] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
10 rows in set (0.00 sec)
```
2024-02-28 13:07:25 +08:00
47d112e4e6 [fix](MySQL) implement SHOW CHARSET statement. (#31389) 2024-02-28 13:07:23 +08:00
747faeed17 [Enhancement](group commit) optimize some group commit code (#31392)
This PR optimizes some of the logic related to group commit:
1. Improved the error handling when there is insufficient WAL space during import.
2. Accounted for cases where the content length is negative during import.
3. Added missing error log printing in `group_commit_mgr.cpp`.
2024-02-28 13:05:57 +08:00
2f6251ccde [pipelineX](refactor) remove source state from operator functions (#31435) 2024-02-28 13:05:57 +08:00
7b3377d474 [fix](Nereids) let with methods of plans use correct logical properties (#31447) 2024-02-28 13:05:57 +08:00
37590f1778 Make sure external table fetched dbId before call getRowCount. (#31379) 2024-02-28 13:05:57 +08:00
41e31ee333 creat hdfs fs with it's resource id (#31505) 2024-02-28 11:33:34 +08:00
eb0416032b [feature](multi-catalog)support hms catalog create and drop table/db (#30198) (#31499)
1. rename old create/drop table to add/removeMemoryTable
2. add new create/drop table/db method
3. support hms catalog create/drop table/db

(cherry picked from commit b2e869c7414c68186de8d43b324ae736d7cc3463)
2024-02-28 09:33:54 +08:00