database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
18,429 Commits 1 Branch 0 Tags
3cb0deae9ceef92c71c35444c495bc5b94de6c7c
Commit Graph

8289 Commits

Author SHA1 Message Date
amory
93a934e775 [Improve](map) support map cast with map literal and implicate nested scala cast (#26126) 2023-11-02 09:56:42 -05:00
jakevin
e307884e57 [enhancement](Nereids): GroupPlan don't generate ObjectId (#26315) 2023-11-02 21:35:48 +08:00
walter
d9621404b3 [improvement](fe) Add more log about add/remove frontends (#26306) 2023-11-02 20:19:51 +08:00
xy720
a5ef90dacc [enhancement](recover) support skipping missing version in select by session variable (#25654) 2023-11-02 20:01:51 +08:00
luozenglin
89cf828f03 [enhancement](udf) add the switch enable_java_udf to control the creation and use of java_udf. (#26213) 2023-11-02 19:33:17 +08:00
谢健
c46fa339d0 [feature](Nereids): make count() avaliable for all data type (#26180) 
make count() avaliable for all data type
 2023-11-02 17:20:24 +08:00
Gabriel
dd8bcc831c [keyword](decimalv2) Add DecimalV2 keyword (#26283) 2023-11-02 16:27:12 +08:00
Xinyi Zou
f1362e0d1e [fix](planner) Fix sample partition table (#25912) 
In the past, two conditions needed to be met when sampling a partitioned table: 1. Data is evenly distributed between partitions; 2. Data is evenly distributed between buckets. Finally, the number of sampled rows in each partition and each bucket is the same.

Now, sampling will be proportional to the number of partitioned and bucketed rows.
 2023-11-02 15:52:48 +08:00
Pxl
3004dbbc78 [Bug](materialized-view) SelectMaterializedIndexWithAggregate do not … (#26192) 
SelectMaterializedIndexWithAggregate do not change plan when match base index
 2023-11-02 14:12:22 +08:00
Qi Chen
3ce1bdae84 [Feature](Nereids) Add explain tree functionality. (#25446) 
Add explain tree functionality which contains fragment ids for profile analysis.
```
mysql> explain tree select sum(l_orderkey) from lineitem, orders where l_orderkey = o_orderkey;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                                                                   |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| [414]:[414: ResultSink]||[Fragment: 0]||VRESULT SINK||   MYSQL_PROTOCAL||                                                                                                                                         |
| --[414]:[414: VAGGREGATE (merge finalize)]||[Fragment: 0]||cardinality=1||                                                                                                                                        |
| ----[411]:[411: VEXCHANGE]||[Fragment: 0]||offset: 0||                                                                                                                                                            |
| ------[411]:[411: DataStreamSink]||[Fragment: 1]||STREAM DATA SINK||  EXCHANGE ID: 411||  UNPARTITIONED                                                                                                           |
| --------[408]:[408: VAGGREGATE (update serialize)]||[Fragment: 1]||cardinality=1||                                                                                                                                |
| ----------[402]:[402: VHASH JOIN]||[Fragment: 1]||join op: INNER JOIN(PARTITIONED)[]||cardinality=593,132,346||                                                                                                   |
| ------------[392]:[392: VEXCHANGE]||[Fragment: 1]||offset: 0||                                                                                                                                                    |
| --------------[392]:[392: DataStreamSink]||[Fragment: 2]||STREAM DATA SINK||  EXCHANGE ID: 392||  HASH_PARTITIONED                                                                                                |
| ----------------[386]:[386: VHIVE_SCAN_NODE]||[Fragment: 2]||table: lineitem||inputSplitNum=144, totalFileSize=16632158739, scanRanges=144||partition=1/1||cardinality=600037902, numNodes=1||pushdown agg=NONE|| |
| ------------[399]:[399: VEXCHANGE]||[Fragment: 1]||offset: 0||                                                                                                                                                    |
| --------------[399]:[399: DataStreamSink]||[Fragment: 3]||STREAM DATA SINK||  EXCHANGE ID: 399||  HASH_PARTITIONED                                                                                                |
| ----------------[393]:[393: VHIVE_SCAN_NODE]||[Fragment: 3]||table: orders||inputSplitNum=48, totalFileSize=3984213353, scanRanges=48||partition=1/1||cardinality=150000000, numNodes=1||pushdown agg=NONE||      |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
12 rows in set (0.04 sec)
```
 2023-11-02 14:06:24 +08:00
zhangdong
fb622800aa [fix](mtmv)Compatible old mtmv metadata (#26193) 
make sure user can upgrade from v2.0 to master
 2023-11-02 12:04:04 +08:00
Jibing-Li
80f654ec2a [Fix](statistics)Fix analyze min max sql syntax error. #26240 2023-11-02 09:22:32 +08:00
jakevin
6010be88bd [enhancement](Nereids): optimize GroupExpressionMatching (#26196) 2023-11-01 19:05:08 +08:00
jakevin
502f5778f4 [enhancement](Nereids): set maxJoinNumBushyTree = 8 (#26173) 2023-11-01 19:04:49 +08:00
TengJianPing
1c2831e06f [fix](spill) disable spill of sort and agg for now to avoid diisk overflow (#26209) 2023-11-01 17:20:26 +08:00
jakevin
683832230c [Performance](Nereids): pass ConnectContext to avoid ThreadLocal.get() (#26165) 2023-11-01 16:16:42 +08:00
JingDas
1770224322 [fix](planner) Fix decimal precision and scale wrong when create table like (#25802) 
Use field datatype such as decimal(10, 0) to create table like. Because the scale is 0, the precision and scale will lost when create table like done. this will fix the bug.

**Before fix, create table with following SQL**:
CREATE TABLE IF NOT EXISTS db_test.table_test
(
    `name` varchar COMMENT "1m size",
    `id` SMALLINT COMMENT "[-32768, 32767]",
    `timestamp0` decimal null comment "c0",
    `timestamp1` decimal(38, 0) null comment "c1"
)
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES ('replication_num' = '1');

**and Then run**
CREATE TABLE db_test.table_test_like LIKE db_test.table_test
SHOW CREATE TABLE db_test.table_test_like;

the field `timestamp1` will be decimal(9, 0), it's wrong. this will fix it.
 2023-11-01 03:11:43 -05:00
zhangdong
5efc6c4233 [fix](auth)Fix can not insert overwrite who have load priv (#25711) 
Fix can not insert overwrite who have load priv
 2023-11-01 15:21:16 +08:00
HB
6de1cc7a0a [fix](udf)java udf does not support overloaded evaluate method (#22681) 
Co-authored-by: morningman <morningman@163.com>
 2023-11-01 15:05:37 +08:00
AKIRA
268c69971d [fix](stats) Store max/min by base64 2023-11-01 14:31:35 +08:00
wangbo
eaed0dea5e [refactor](executor)Add % suffix for cpu_hard_limit #26174 2023-11-01 10:27:48 +08:00
morrySnow
18dabe7386 [fix](compile) fe compile failed when generate doc and FE UT failed (#26164) 
1. FE could not compile because below error. Intro by PR #25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - #25951
> - #26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR #25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (#26145)"
> This reverts commit 8d7abf60f94d2d1208b71e96b9290ea02122b8d8.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (#26130)"
> This reverts commit 19122b55cd95af097b4ef7b6eb809f37db29765f.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (#26084)"
> This reverts commit 0d956e90cf920039b8baa79c170a298be56a128d.
 2023-10-31 20:50:44 -05:00
yujun
ac7678926f [improvement](tablet clone) add config for decommission tablet wait time (#26000) 2023-10-31 20:36:26 +08:00
Calvin Kirs
08c78a1135 [Feature](Job)Support manual and refactor some execution logic (#26082) 
Supports manually triggered JOBs and Tasks
Optimize JOB&TASK display logic
Refactor the executor to support context passing
 2023-10-31 20:35:55 +08:00
wuwenchi
b98744ae90 [Bug](iceberg)fix read partitioned iceberg without partition path (#25503) 
Iceberg does not require partition values to exist on file paths, so we should get the partition value from `PartitionScanTask.partition`.
 2023-10-31 18:09:53 +08:00
Mryange
b137f03921 [feature](profile) add MaxActiveTime and add exectime in pipelineX (#26148) 2023-10-31 17:53:52 +08:00
Pxl
271b074f42 Bug](view) make forbiddenMVRewrite work on InlineViewRef (#26162) 
make forbiddenMVRewrite work on InlineViewRef
 2023-10-31 17:30:59 +08:00
Calvin Kirs
6f316b117f [Fix](Checkpoint)AnalysisMgrV2 does not implement the corresponding read and write interface (#26161) 2023-10-31 17:24:35 +08:00
slothever
994db8b400 [fix](multi-catalog)add the FAQ for Aliyun DLF and add the fs.xx.impl check (#25594) 
1. add the FAQ for Aliyun DLF，include the DLF configuration and Jindo SDK locations
2. add the fs.xx.impl check when load cache files
 2023-10-31 17:09:52 +08:00
starocean999
958c7fdfd9 [fix](planner)should keep at least one slot materialized in agg node (#26116) 2023-10-31 16:52:45 +08:00
minghong
a6e010f722 [opt](nereids) use rowcount to compute runtime filter size if ndv is not available (#26022) 2023-10-31 16:52:17 +08:00
zhangdong
a259ae9417 [fix](mtmv)add mtmvJobManager to DEPRECATED_MODULE_NAMES (#26137) 
Inherited from：26041
 2023-10-31 16:28:43 +08:00
jakevin
19122b55cd [enhancement](Nereids): optimize GroupExpressionMatching (#26130) 
1. pattern can't be SubTreePattern in CBO phase.
2. optimize getInputSlotExprId()
 2023-10-31 15:47:13 +08:00
morrySnow
111b8e2b4f [minor](planner) remove len on integral type toSql (#25951) 2023-10-31 02:37:08 -05:00
zhiqiang
feefebe0da [chore](log opt) opt mysql load log (#26149) 2023-10-31 14:59:52 +08:00
morrySnow
95c765f5fe [fix](Nereids) values list could not process only one column (#26123) 2023-10-31 01:58:54 -05:00
LiBinfeng
ea1e8fa15d [Fix](Nereids) fix create table as select of view with unknowed length character type (#25471) 
Problem:
when create table as select from a view with unknown length character type, be would return an error of inserting data failed

Example:
doris/regression-test/suites/ddl_p0/test_ctas.groovy

Reason & Solved:
BE can not derive varchar length automaticly so FE should tell BE to maximize the size of varchar type
 2023-10-31 01:50:28 -05:00
zhiqiang
0449a240f4 [Fix](from_unixtime) Keep consistent with MySQL & bug fix (#25966) 
Bug fix: implicit convert from int32 -> int64 makes negative time stamp valid, so change signature to int64
Consistent: keep consistent with mysql.
 2023-10-31 14:31:24 +08:00
Pxl
8d7abf60f9 [Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan when match ba… (#26145) 
* SelectMaterializedIndexWithAggregate do not change plan when match base index

* forbid table withh alias
 2023-10-31 14:11:08 +08:00
Guangdong Liu
588d4c150f [bugfix](DateTimeV2Literal) The start and end dates of daylight saving time vary from year to year, which can lead to different test results at different times (#26083) 
The start and end dates of daylight saving time vary depending on the year. Daylight saving time usually starts on the last Sunday in March and ends on the last Sunday in October. During this time, the offset for Europe/London is UTC+1. Outside of daylight saving time, the Europe/London offset is UTC+0.
 2023-10-31 13:39:38 +08:00
Gabriel
809510f8b2 [bug](udf) Fix method invoking (#26131) 2023-10-31 11:46:14 +08:00
Jibing-Li
78204f7c92 [Fix](statistics)Fix external couldn't analyze database bug (#26025) 2023-10-31 11:32:47 +08:00
谢健
b3f31f9e1a [enhancement](Nereids): speedup graphsimplifier (#26066) 
1. fix some bugs in graphsimplifier
2. remove some time costed code
3. add shape check for graphsimplifier
 2023-10-30 20:13:13 +08:00
zhangdong
2a74d9a8c8 [feature](mtmv)(1)remove old mtmv code (#26041) 
remove old mtmv code,we will implement mtmv in a new way
 2023-10-30 19:49:45 +08:00
谢健
f97c40d0e1 [fix](Nereids) fix plan shape of query64 unstable (#26012) 
don't remove the physical plan after optimizing the plan in dphyper.
 2023-10-30 19:49:28 +08:00
morrySnow
f70dfe27d0 [fix](Nereids) init comment as null when create struct field from catalog type (#26105) 2023-10-30 06:26:09 -05:00
Gabriel
6eb2cb2d48 [date](parser) Support DateV1 keyword (#25414) 2023-10-30 18:39:22 +08:00
谢健
844b7c8cba [enhancement](Nereids): check stats unreliable when deriving stats (#26103) 
check stats unreliable when deriving stats
 2023-10-30 17:21:42 +08:00
Yulei-Yang
3a954cd1aa [fix](function)return NULL rather than 'null' if path not found (#25880) 
fix json_extract not return NULL but null
 2023-10-30 14:26:44 +08:00
jakevin
0d956e90cf [Performance](Nereids): optimize GroupExpressionMatching (#26084) 2023-10-30 14:05:25 +08:00