Commit Graph

17549 Commits

Author SHA1 Message Date
f1e14722d8 [docs](docs) Update 2.0.5 Release Note of EN & CN Version (#31506) 2024-02-28 13:08:36 +08:00
d0a8a30998 [improvement](iceberg/paimon)add show table stats (#31473) 2024-02-28 13:08:36 +08:00
5bbe9f7b40 Fix replay binlog gc when not found db binlog (#31463)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2024-02-28 13:07:47 +08:00
5824f8a4bc [Feat](nereids) support multi-leading (#30379)
support multi-leading in each query block
2024-02-28 13:07:47 +08:00
c5e6e3d45f [fix](docs)Correcting spelling errors,add isolated file to siderbars. (#31409)
Correcting spelling errors,add isolated file to siderbars
2024-02-28 13:07:47 +08:00
e86cc7e8e8 [chore](log) reduce a lot inject debug point log #31474 2024-02-28 13:07:47 +08:00
ab00c7012b [fix] Fix the incorrect class name in the getLogger method call of MysqlTable. (#31465) 2024-02-28 13:07:47 +08:00
Pxl
3acfda413b [Chore](function) remove unused check on count function (#31400) 2024-02-28 13:07:47 +08:00
7f566f9365 Reset report_workload_runtime_status to optional (#31479) 2024-02-28 13:07:47 +08:00
04de7aed40 [fix](case) fix testcase test_decimal256_index #31486 2024-02-28 13:07:47 +08:00
a371a10603 [fix](Nereids) let time type coercion same with legacy planner (#31472) 2024-02-28 13:07:47 +08:00
c0754583cb [opt](plsql) Fix procedure key compatibility (#31445)
use dbId replace dbName, because dbName may be renamed by Alter.
procedure key add package name (only reserved, currently no plans to support package)
Optimize procedure create and exception
2024-02-28 13:07:47 +08:00
9ffcf48cce [enhancement](Nereids) Support show process time and process steps by explain statement (#31339)
## Proposed changes

1. show process time when execute `explain plan xxx` by nereids
2. add `explain xxx plan process select ...` statement to show the process of the plan, not support show memo shape (physical plan) currently

example:
show process time:
```
mysql> explain plan select * from tt;
+---------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                               |
+---------------------------------------------------------------------------------------------------------------+
| ========== PARSED PLAN (time: 3ms) ==========                                                                 |
| UnboundResultSink[3] (  )                                                                                     |
| +--LogicalProject[2] ( distinct=false, projects=[*], excepts=[] )                                             |
|    +--LogicalCheckPolicy (  )                                                                                 |
|       +--UnboundRelation ( id=RelationId#0, nameParts=tt )                                                    |
|                                                                                                               |
| ========== ANALYZED PLAN (time: 6ms) ==========                                                               |
| LogicalResultSink[11] ( outputExprs=[id#0, name#1] )                                                          |
| +--LogicalProject[9] ( distinct=false, projects=[id#0, name#1], excepts=[] )                                  |
|    +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON ) |
|                                                                                                               |
| ========== REWRITTEN PLAN (time: 0ms)==========                                                               |
| LogicalResultSink[11] ( outputExprs=[id#0, name#1] )                                                          |
| +--LogicalProject[9] ( distinct=false, projects=[id#0, name#1], excepts=[] )                                  |
|    +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON ) |
|                                                                                                               |
| ========== OPTIMIZED PLAN (time: 2ms) ==========                                                              |
| PhysicalResultSink[56] ( outputExprs=[id#0, name#1] )                                                         |
| +--PhysicalDistribute[53]@1 ( stats=2, distributionSpec=DistributionSpecGather )                              |
|    +--PhysicalProject[50]@1 ( stats=2, projects=[id#0, name#1] )                                              |
|       +--PhysicalOlapScan[tt]@0 ( stats=2 )                                                                   |
+---------------------------------------------------------------------------------------------------------------+
21 rows in set (0.01 sec)
```

explain plan process:
```
mysql> explain plan process select * from tt\G
*************************** 1. row ***************************
  Rule: BINDING_RELATION
Before: UnboundResultSink[8] (  )
+--LogicalProject[7] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalCheckPolicy (  )
      +--UnboundRelation ( id=RelationId#0, nameParts=tt )
 After: UnboundResultSink[11] (  )
+--LogicalProject[10] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalCheckPolicy (  )
      +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 2. row ***************************
  Rule: CHECK_ROW_POLICY
Before: UnboundResultSink[15] (  )
+--LogicalProject[14] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalCheckPolicy (  )
      +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: UnboundResultSink[17] (  )
+--LogicalProject[16] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 3. row ***************************
  Rule: BINDING_PROJECT_SLOT
Before: UnboundResultSink[22] (  )
+--LogicalProject[21] ( distinct=false, projects=[*], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: UnboundResultSink[23] (  )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 4. row ***************************
  Rule: BINDING_RESULT_SINK
Before: UnboundResultSink[26] (  )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[25] ( outputExprs=[id#0, name#1] )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 5. row ***************************
  Rule: ELIMINATE_UNNECESSARY_PROJECT
Before: LogicalResultSink[25] ( outputExprs=[id#0, name#1] )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[27] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 6. row ***************************
  Rule: PRUNE_EMPTY_PARTITION
Before: LogicalResultSink[29] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[30] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
*************************** 7. row ***************************
  Rule: MATERIALIZED_INDEX_SCAN
Before: LogicalResultSink[36] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[37] ( outputExprs=[id#0, name#1] )
+--LogicalProject[35] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalProject[34] ( distinct=false, projects=[id#0, name#1], excepts=[] )
      +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
*************************** 8. row ***************************
  Rule: MERGE_PROJECTS
Before: LogicalResultSink[40] ( outputExprs=[id#0, name#1] )
+--LogicalProject[39] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalProject[34] ( distinct=false, projects=[id#0, name#1], excepts=[] )
      +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[41] ( outputExprs=[id#0, name#1] )
+--LogicalProject[38] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
*************************** 9. row ***************************
  Rule: ELIMINATE_UNNECESSARY_PROJECT
Before: LogicalResultSink[42] ( outputExprs=[id#0, name#1] )
+--LogicalProject[38] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[43] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
*************************** 10. row ***************************
  Rule: REWRITE_CTE_CHILDREN
Before: LogicalResultSink[25] ( outputExprs=[id#0, name#1] )
+--LogicalProject[20] ( distinct=false, projects=[id#0, name#1], excepts=[] )
   +--LogicalOlapScan ( qualified=test.tt, indexName=<index_not_selected>, selectedIndexId=10361, preAgg=ON )
 After: LogicalResultSink[43] ( outputExprs=[id#0, name#1] )
+--LogicalOlapScan ( qualified=test.tt, indexName=tt, selectedIndexId=10361, preAgg=ON )
10 rows in set (0.00 sec)
```
2024-02-28 13:07:25 +08:00
47d112e4e6 [fix](MySQL) implement SHOW CHARSET statement. (#31389) 2024-02-28 13:07:23 +08:00
747faeed17 [Enhancement](group commit) optimize some group commit code (#31392)
This PR optimizes some of the logic related to group commit:
1. Improved the error handling when there is insufficient WAL space during import.
2. Accounted for cases where the content length is negative during import.
3. Added missing error log printing in `group_commit_mgr.cpp`.
2024-02-28 13:05:57 +08:00
2f6251ccde [pipelineX](refactor) remove source state from operator functions (#31435) 2024-02-28 13:05:57 +08:00
7b3377d474 [fix](Nereids) let with methods of plans use correct logical properties (#31447) 2024-02-28 13:05:57 +08:00
37590f1778 Make sure external table fetched dbId before call getRowCount. (#31379) 2024-02-28 13:05:57 +08:00
41e31ee333 creat hdfs fs with it's resource id (#31505) 2024-02-28 11:33:34 +08:00
eb0416032b [feature](multi-catalog)support hms catalog create and drop table/db (#30198) (#31499)
1. rename old create/drop table to add/removeMemoryTable
2. add new create/drop table/db method
3. support hms catalog create/drop table/db

(cherry picked from commit b2e869c7414c68186de8d43b324ae736d7cc3463)
2024-02-28 09:33:54 +08:00
82fd3af54b [chore](log) change merge-on-write correctness check log to VLOG_NOTICE (#31414) (#31467) 2024-02-27 23:36:24 +08:00
39a8db27f2 [fix](mtmv)TVF Query JOB Concurrent Reading and Writing Causes Exception #31422 2024-02-27 16:06:26 +08:00
6b4a756837 [Fix] Only datetime and datetimev2 types can use current_timestamp as column default value (#31395)
for this kind of sql:

create table test_default10(
  a int, 
  b varchar(100) default current_timestamp
)
distributed by hash(a)
properties('replication_num'="1");

add check:
 Types other than DATETIME and DATETIMEV2 cannot use current_timestamp as the default value
2024-02-27 16:06:26 +08:00
fb1e08ee8c [doc](colocate) Fix colocate api doc inconformity with code (#31430)
```
if ("POST".equalsIgnoreCase(method)) {
  colocateIndex.markGroupUnstable(groupId, "mark unstable via http api", true);
} else if ("DELETE".equalsIgnoreCase(method)) {
  colocateIndex.markGroupStable(groupId, true, null);
}
```
2024-02-27 16:06:26 +08:00
378ced72db [chore](Nereids) more reasonable parse select list only query (#31346) 2024-02-27 16:06:26 +08:00
f039ec8cfb [debug](Variant) sanitize variant type and column in find_and_set_leave_value (#31436) 2024-02-27 13:58:13 +08:00
434396b456 [docs](docs) Update docs' sidebar of Dev version (#31444) 2024-02-27 13:58:13 +08:00
735184b264 [test](auto partition)increase max_auto_partition_num configure value of auto partition (#31410)
* [test](auto partition)increase auto partition max_auto_partition_num configure value

* fix bug
2024-02-27 13:58:13 +08:00
4fd00e3ad7 [Fix](test)Fix workload group query queue regression test #31419 2024-02-27 10:12:53 +08:00
4b0d6716dc [Fix](be)Fix gcc compile failed #31431 2024-02-27 10:12:53 +08:00
19196ae1cf [doc](nereids) Optimize query rewerite by materialzied view doc (#31420)
* [doc](nereids) Optimize the query rewrite by materialized view doc

* [doc](nereids) add more
2024-02-27 10:12:53 +08:00
3b093cabd1 Fix building issue in be on ubuntu with test enabled. (#31407)
Co-authored-by: tangye <tangye@bestpay.com.cn>
2024-02-27 10:12:44 +08:00
1fbf32ead2 [enhancement](pipelinex) limit add_child to and dependency to avoid error (#31394)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-02-27 10:12:36 +08:00
6e62017ed5 [fix](scanner) allocated_bytes should be called after success (#31428)
allocated_bytes should be called after success
2024-02-27 10:12:36 +08:00
481d94c3fc [feature](nereids) deal the slots that appear both in agg func and grouping sets (#31318)
this PR support slot appearing both in agg func and grouping sets.
sql like below:
select sum(a) from t group by grouping sets ((a)); 

Before this PR, Nereids throw exception like below:
col_int_undef_signed cannot both in select list and aggregate functions when using GROUPING SETS/CUBE/ROLLUP, please use union instead.

This PR removes the restriction and supports this situation.
2024-02-27 10:12:33 +08:00
dd229b77b1 [fix](inverted index)Remove the strong check for parser when creating a table with inverted index (#31391) 2024-02-27 10:12:33 +08:00
c34639245e [Improvement](executor)add remote scan thread pool (#31376)
* add remote scan thread pool

* +1
2024-02-27 10:12:33 +08:00
1127b0065a [Improment](executor)Add scanbytes/scanrows condition (#31364)
* Add scanbytes/scanrows condition

* fix reg
2024-02-27 10:12:33 +08:00
b8fe620ba3 Fix cgroup cpu controller file leak (#31398) 2024-02-27 10:12:33 +08:00
f163d56a98 [feature](function) support sequence function(alias of array_range), enhance both to handle datetimev2 (#30823) 2024-02-27 10:12:19 +08:00
22b6434054 [feature](doris compose) Add create cloud cluster (#31315) 2024-02-27 10:12:19 +08:00
3cee6c6722 [fix](function) fix unexpected be core in string search function (#31312)
Fix be core in multi_match_any/multi_search_all_positions functions.
2024-02-27 10:12:18 +08:00
35333d7a77 [opt](scanner) scan enough blocks in each scan task (#31277) 2024-02-27 10:12:18 +08:00
0f38769102 [fix](scan) Fix missing shared tablet header lock (#31433) 2024-02-26 23:39:10 +08:00
5d4a2d93a6 [feature](nereids) support join with joinRelation (#30909) 2024-02-26 19:07:11 +08:00
bc41033b5d [doc](community) modify the release verify doc (#31385) 2024-02-26 19:07:11 +08:00
d6f143b7d5 [doc](job)add job doc sidebars (#31319)
* add job doc sidebars

* fix
2024-02-26 19:07:11 +08:00
60c002ba91 [Docs](Job)Add job Docs (#28186)
* [Docs](Job)Add job Docs

* format

* Fix job error
2024-02-26 19:07:11 +08:00
Pxl
fcea2b964e [Chore](materialized-view) forbid create mv have calculations included outside aggregate functions (#31336)
forbid create mv have calculations included outside aggregate functions
2024-02-26 19:07:11 +08:00
b1fc0ebbe7 [improvement](iceberg/paimon)support estimate row count (#31204)
Get the number of rows evaluated for iceberg and paimon.
2024-02-26 19:07:10 +08:00