database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
13,842 Commits 1 Branch 0 Tags
136973d4faefe8bb05914033bc245b2eaa355f9e
Commit Graph

13842 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kang
136973d4fa [fix](testcase) add state check for ADD INDEX before BUILD INDEX to avoid table state not normal (#25038) 2023-10-05 22:15:54 +08:00
Kaijie Chen
c2b46e4df7 [fix](move-memtable) exclude rpc memory in flush mem-tracker (#24722) 2023-10-05 22:10:53 +08:00
Kaijie Chen
db6c16058a [improve](move-memtable) always share load streams (#24763) 2023-10-05 22:09:59 +08:00
bobhan1
d1f4d69032 [regression-test](merge-on-write) Add cases for partial update using insert statement with schema change (#24902) 2023-10-05 22:09:22 +08:00
HappenLee
93eedaff62 [opt](function) Use Dict to opt the function of time_round (#25029) 
Before：

select hour_floor(`@timestamp`, 7) as t, count() as cnt from httplogs_date group by t order by t limit 10;
+---------------------+--------+
| t                   | cnt    |
+---------------------+--------+
| 1998-04-30 21:00:00 |    324 |
| 1998-05-01 04:00:00 | 286156 |
| 1998-05-01 11:00:00 | 266130 |
| 1998-05-01 18:00:00 | 483765 |
| 1998-05-02 01:00:00 | 276706 |
| 1998-05-02 08:00:00 | 169945 |
| 1998-05-02 15:00:00 | 223593 |
| 1998-05-02 22:00:00 | 272616 |
| 1998-05-03 05:00:00 | 188689 |
| 1998-05-03 12:00:00 | 184405 |
+---------------------+--------+
10 rows in set (3.39 sec)
after:

select hour_floor(`@timestamp`, 7) as t, count() as cnt from httplogs_date group by t order by t limit 10;
+---------------------+--------+
| t                   | cnt    |
+---------------------+--------+
| 1998-04-30 21:00:00 |    324 |
| 1998-05-01 04:00:00 | 286156 |
| 1998-05-01 11:00:00 | 266130 |
| 1998-05-01 18:00:00 | 483765 |
| 1998-05-02 01:00:00 | 276706 |
| 1998-05-02 08:00:00 | 169945 |
| 1998-05-02 15:00:00 | 223593 |
| 1998-05-02 22:00:00 | 272616 |
| 1998-05-03 05:00:00 | 188689 |
| 1998-05-03 12:00:00 | 184405 |
+---------------------+--------+
10 rows in set (2.19 sec)
 2023-10-04 23:34:24 +08:00
meiyi
4ce5213b1c [fix](insert) Fix test_group_commit_stream_load and add more regression in test_group_commit_http_stream (#24954) 2023-10-03 20:56:24 +08:00
yujun
c298b1ca1a [fix](timezone) fix parse timezone when include GMT or time zone short ids (#25032) 2023-10-03 20:53:16 +08:00
zy-kkk
6e836fe381 [fix](jdbc catalog) fix jdbc catalog read bitmap data crash (#25034) 2023-10-03 20:52:47 +08:00
Guangdong Liu
d77d56b2c6 [fix](regression test) fix ci error (#25036) 2023-10-03 20:52:00 +08:00
amory
10f0c63896 [FIX](complex-type) fix agg table with complex type with replace state (#24873) 
fix agg table with complex type with replace state
 2023-10-03 16:32:58 +08:00
HappenLee
f8a3034dca [Opt](performance) refactor and opt time round floor function (#25026) 
refactor and opt time round floor function
 2023-10-01 11:51:26 +08:00
AlexYue
113f0684a6 [chore](case) Simplify cold heat separation case (#25010) 2023-09-30 15:35:42 +08:00
bobhan1
642e5cdb69 [Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly (#23395) 2023-09-29 22:38:52 +08:00
HHoflittlefish777
8040ba7fb9 [Fix](regression-test) Fix stream load 2pc case (#24925) 2023-09-28 22:34:59 +08:00
Siyang Tang
2c25e0a681 [test](load) add more s3 load regression test cases (#24906) 2023-09-28 22:01:36 +08:00
minghong
4c94820ff9 [opt](nereids) adjust column stats in filter estimation (#24973) 
TPCDS before
query4  9335    8113    8070    8070
query13 3104    1386    1385    1385
query18 1704    1216    1151    1151
query48 840     840     839     839
query61 435     379     383     379
query71 715     570     579     570
query85 2822    2627    2612    2612
query88 1897    1816    1793    1793
Total cold run time: 20852 ms
Total hot run time: 16799 ms

after:
query4  9610    8287    8249    8249
query13 1721    1013    1042    1013
query18 1585    1186    1155    1155
query48 789     777     778     777
query61 384     387     381     381
query71 713     610     584     584
query85 2020    1867    1843    1843
query88 1859    1812    1805    1805
Total cold run time: 18681 ms
Total hot run time: 15807 ms
 2023-09-28 21:34:17 +08:00
morrySnow
8eaf0d3a4b [fix](Nereids) ctas varchar length should set to max except column from slot (#25003) 2023-09-28 21:32:33 +08:00
zhengyu
d23bedf170 [fix](single-replica-load) fix duplicated done run in request_slave_tablet_pull_rowset (#25013) 
BE will crash because done run twice when try_offer() failed in
request_slave_tablet_pull_rowset.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
 2023-09-28 21:08:18 +08:00
starocean999
5dd70b8a25 [fix](planner) createColumnAndViewDefs method use wrong analyzer (#25005) 2023-09-28 06:59:04 -05:00
Lijia Liu
864a0f9bcb [opt](pipeline) Make pipeline fragment context send_report asynchronized (#23142) 2023-09-28 17:55:53 +08:00
JingDas
230b7bd15e [test](nereids) Add some tests for PushFilterInsideJoin and FindHashConditionForJoin rule (#24550) 2023-09-28 03:45:05 -05:00
morrySnow
bf4fb32487 [minor](catalog) remove useless compatibilityMatrix in catalog PrimitiveType (#24999) 2023-09-28 02:57:59 -05:00
zhannngchen
2ec50dcfc7 [log](compaction) add more stats for compaction log (#24984) 2023-09-28 15:29:15 +08:00
谢健
a574f29d76 [enhancement](Nereids): use enforcer to choose the n-th plan (#22929) 2023-09-28 15:16:24 +08:00
morrySnow
b50c1448df [fix](Nereids) should not replace slot by Alias when do NormalizeSlot (#24928) 
when we do NormalizeToSlot, we pushed complex expression and only remain
slot of it. When we do this, we collect alias and their child and
compute its child in bottom project, remain the result slot in current
node. for example

Window(max(...), c1 as a1)

after normalization, we get

Window(max(...), a1)
+-- Project(..., c1 as a1)

But, in some cases, we remove some SlotReference by mistake, for example

Window(max(...), c1, c1 as a1)

after normalization, we get

Window(max(...), a1)
+-- Project(..., c1 as a1)

we lost the SlotReference c1. This PR fix this problem. After this Pr,
we get

Window(max(...), c1, a1)
+-- Project(..., c1, c1 as a1)
 2023-09-28 14:51:08 +08:00
Calvin Kirs
377554ee1c [Fix](Job)Job Task does not display error message (#24897) 2023-09-28 14:47:12 +08:00
zhangstar333
b6babf3af4 [pipelineX](sink) support jdbc table sink (#24970) 
* [pipelineX](sink) support jdbc table sink
 2023-09-28 14:39:32 +08:00
xzj7019
e863cfe5c7 [fix](nereids) fix multi window projection issue temporarily (#24912) 
Current multi-window plan generation has problem on the project sequence, for example:

+--LogicalWindow ( windowExpressions=[avg(sum_sales#115) WindowSpec(...) AS `avg_monthly_sales`#116, rank() WindowSpec(...) AS `rn`#117], ...)
and correspond physical plan is:

+--PhysicalWindow[6572]@16 ( windowFrameGroup=(Funcs=[avg(sum_sales#115) WindowSpec(...) AS `avg_monthly_sales`#116], ... )
    +--PhysicalWindow[6568]@29 ( windowFrameGroup=(Funcs=[rank() WindowSpec(...) AS `rn`#117], ...] )
If the final plan is generated as following:

MultiCastDataSinks
STREAM DATA SINK
  EXCHANGE ID: 20
  HASH_PARTITIONED: rn[#208], i_brand[#202], cc_name[#203], i_category[#201]
Before we eventually resolve the multi-window issue, we add a projection as following and force a mapping but this will not cover all potential problems.

MultiCastDataSinks
STREAM DATA SINK
  EXCHANGE ID: 20
  HASH_PARTITIONED: rn[#219], i_brand[#213], cc_name[#214], i_category[#212]
  PROJECTIONS: i_category[#184], i_brand[#185], cc_name[#186], d_year[#187], d_moy[#188], sum_sales[#189], avg_monthly_sales[#191], rn[#190]
  PROJECTION TUPLE: 20
 2023-09-28 14:33:00 +08:00
Calvin Kirs
f5c38b29a5 [Improve](Load)Change the response label prefix of Update and Delete to the corresponding operations (#24996) 
Doris Whether it is insert, delete, or update, the label prefix is insert, which may confuse users.

Change
Add Update and Delete label prefix

Test

mysql>  insert into t2 (id,id_str) values (2,'test2');
Query OK, 1 row affected (0.09 sec)
{'label':'insert_b16405a387f14bfa_947dc9b2217ee3df', 'status':'VISIBLE', 'txnId':'17023'}

mysql>  insert into t1 (id,id_str) values (2,'test2');
Query OK, 1 row affected (0.09 sec)
{'label':'insert_c3acdf63bf94e87_ad65a2dca88f5576', 'status':'VISIBLE', 'txnId':'17025'}

mysql> update t2 set id_str='update2';
Query OK, 2 rows affected (5.27 sec)
{'label':'update_903a88c8defe41d5_a7fca85159c84e50', 'status':'VISIBLE', 'txnId':'17026'}


mysql> delete from  t2  where id =2;
Query OK, 0 rows affected (5.56 sec)
{'label':'delete_1ca419aa-b7a2-41f6-9cbd-e14f4c7517f4', 'status':'VISIBLE', 'txnId':'17028'}

mysql> delete from t1 where t1.id in (select id from t2);
Query OK, 1 row affected (4.41 sec)
{'label':'delete_7e2ae75fee9a42b7_9322d4ae8b80a28b', 'status':'VISIBLE', 'txnId':'17034'}
 2023-09-28 14:32:35 +08:00
Gabriel
b35171b582 [pipelineX](bug) fix distinct streaming agg (#24995) 2023-09-28 14:01:26 +08:00
Hu Yanjun
5ffb01a068 [doc](doc)Update doc: delete Z-order (#24955) 2023-09-28 13:45:48 +08:00
zhiqqqq
11d03a3ab0 [thirdyparty] new thirdy party dragonbox (#24979) 2023-09-28 13:42:44 +08:00
jakevin
bf808e9aa6 [fix](Nereids): tolerate DateLike overflow in SQL CAST/CONVERT (#24943) 
- explicit type cast, we need tolerate overflow and convert it to be NULL
- implicit type cast, throw exception
 2023-09-28 12:11:50 +08:00
Gabriel
f0fad61db4 [pipelineX](bug) Fix file scan operator (#24989) 2023-09-28 11:12:27 +08:00
Siyang Tang
188d9ab94e [enhancement](statistics) collect table level loaded rows on BE to make RPC light weight (#24609) 2023-09-28 10:51:50 +08:00
minghong
42207df89f [refactor](nereids)update NormalizeRepeat comments (#24893) 
update NormalizeRepeat comments
 2023-09-28 10:42:16 +08:00
zzzzzzzs
4ff1ab7a4d [fix](regression-test) regenerate test_http_stream_properties.out file (#24946) 2023-09-28 10:39:15 +08:00
Tiewei Fang
21d6f41492 [fix](regresion-test) Fix the problem of occasional failure of test_outfile_exception regression-test case (#24937) 2023-09-28 10:05:43 +08:00
starocean999
584646c054 [improvement](nereids)dphyper GraphSimplifier should consider missed edges when estimating join cost (#21747) 2023-09-28 09:30:57 +08:00
Mryange
430634367a [pipelineX](node)support file scan operator (#24924) 2023-09-27 22:10:43 +08:00
Chenyang Sun
68087f6c82 [fix](json function) Fix the slow performance of get_json_path when processing JSONB (#24631) 
When processing JSONB, automatically convert to jsonb_extract_string
 2023-09-27 21:17:39 +08:00
airborne12
732f821c15 [Fix](inverted index) make parser mode coarse grained by default (#24949) 2023-09-27 21:04:41 +08:00
Liqf
d4e823950a [bug](json)Fix some problems of json function on Nereids (#24898) 
Fix some problems of json_length and json_contains function on Nereids
fix wrong result of json_contains function
Regression test jsonb_p0 to enable Nereids
 2023-09-27 21:01:45 +08:00
meiyi
391a4e29eb [fix](schema) Table column order is changed if add a column and do truncate (#24981) 2023-09-27 20:59:11 +08:00
morrySnow
63b283a848 [fix](Nereids) init Date/DateV2Literal should check non-zero time fields (#24971) 2023-09-27 20:48:36 +08:00
Gabriel
947b116318 [pipelineX](fix) Fix BE crash due to ES scan operator (#24983) 2023-09-27 20:45:38 +08:00
Gabriel
1fb9022d07 [pipelineX](bug) Fix meta scan operator (#24963) 2023-09-27 20:34:47 +08:00
Gabriel
671b5f0a0a [Bug](pipeline) Fix block reusing for union source operator (#24977) 
[CANCELLED][INTERNAL_ERROR]Merge block not match, self:[String], input:[String, Nullable(String), Nullable(String), Nullable(String), Nullable(String), DateV2]
 2023-09-27 19:41:56 +08:00
xzj7019
bb7f8d18a8 [fix](nereids) push down filter through partition topn (#24944) 
support pushing down filter through partition topn if the filter can pass through window.
fix CreatePartitionTopNFromWindow bug which may generate two partition topn unexpectly.
case:
select * from (select c2, row_number() over (partition by c2) as rn from t1) T where rn<=1 and c2 = 1;
before this pr:
| PhysicalResultSink                       |
| --PhysicalDistribute                     |
| ----filter((rn <= 1))                    |
| ------PhysicalWindow                     |
| --------PhysicalQuickSort                |
| ----------PhysicalDistribute             |
| ------------PhysicalPartitionTopN        |
| --------------filter((T.c2 = 1))         |
| ----------------PhysicalPartitionTopN    |
| ------------------PhysicalProject        |
| --------------------PhysicalOlapScan[t1] |
+------------------------------------------+
after:

| PhysicalResultSink                     |
| --PhysicalDistribute                   |
| ----filter((rn <= 1))                  |
| ------PhysicalWindow                   |
| --------PhysicalQuickSort              |
| ----------PhysicalDistribute           |
| ------------PhysicalPartitionTopN      |
| --------------PhysicalProject          |
| ----------------filter((T.c2 = 1))     |
| ------------------PhysicalOlapScan[t1] |
+----------------------------------------+
 2023-09-27 19:38:04 +08:00
morrySnow
00786a3295 [fix](Nereids) could not prune datev1 partition column (#24959) 
because storage engine could not process date comparison predicates.
we convert it to datetime comparison predicates.
however, partition prunner could not process cast(slot) cp literal.
so, we convert back in partition pruner to let it work well.

TODO:
move convert date to datetime in translate stage
and only convert predicates for storage engine.
 2023-09-27 18:41:56 +08:00