database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,529 Commits 1 Branch 0 Tags
54787f51c7ed454feff1663c4e4255c011749421
Commit Graph

62 Commits

Author SHA1 Message Date
abmdocrt
53ca312751 [cherry-pick](branch-2.1) Pick "[Fix](delete) Fix delete on value case due to compaction (#37917)" (#38116) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Pick #37917
 2024-07-19 13:51:39 +08:00
Jerry Hu
8440303b91 [fix](delete) Incorrect precision detection for the decimal type in condition.​ (#37801) (#37904) 
## Proposed changes

pick #37801

For precision like Decimal(7,7), the value "0.1234567" should be
valid(the integer part is 0).

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-16 19:02:02 +08:00
lw112
b55dd6f644 [fix](delete) fix the error message for valid decimal data for 2.1 (#37710) 
## Proposed changes

cherry-pick : #36802

<!--Describe your changes.-->
 2024-07-15 09:54:42 +08:00
abmdocrt
2d5eb55310 [cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add some partial update case (#36385)" (#37382) 
#36385
 2024-07-09 10:52:04 +08:00
Gavin Chou
9c270e5cdf [fix](delete) Fix unrecognized column name delete handler (#32429) (#35742) 
pick doris-master #32429
 2024-05-31 20:41:22 +08:00
morrySnow
11175208df [fix](delete) storage engine delete do not support bitmap (#34710) 2024-05-13 22:07:53 +08:00
morrySnow
c0cca6103b [WIP](test) remove enable_nereids_planner in regression cases (part 2) (#34490) 2024-05-10 22:02:32 +08:00
zhangstar333
a4151e022e [bug](fold) fix fold constant rule can't handle variable expr (#32313) 2024-03-21 14:07:23 +08:00
zhangstar333
3cc409b14f [bug](function) fix date_sub function failed when arg type is datev2 (#30443) 
* [bug](function) fix date_sub function failed when arg type is datev2

* update
 2024-02-04 14:28:38 +08:00
morrySnow
f98f790a80 [chore](test) update delete complex type case to make Nereids happy (#30022) 2024-01-16 20:23:09 +08:00
morrySnow
847898bf26 [fix](Nereids) delete using should support sql without where (#29518) 2024-01-12 11:35:29 +08:00
zhangstar333
f3bbc7b876 [enchancement](delete) fix delete stmt return error with fold on be (#28557) 2024-01-05 11:27:21 +08:00
bobhan1
51bd402b58 [fix](planner) Fix delete statement on MOR unique table (#28968) 
add conditions for mor table removed in #26776
 2023-12-27 12:59:15 +08:00
starocean999
6085a61f1e [chore](test) correct create table statement (#28863) 2023-12-25 10:55:35 +08:00
bobhan1
4300fdce43 [feature](merge-on-write) enable merge-on-write by default again (#28105) 
fix #27188, #28096
 2023-12-14 19:35:52 +08:00
yiguolei
8c79b86f5b Revert "[feature](merge-on-write) enable merge-on-write by default (#27188)" (#28096) 
This reverts commit 00c8bab84de8154052f9d323800b436cd0ad36e5.
 2023-12-07 11:31:36 +08:00
bobhan1
00c8bab84d [feature](merge-on-write) enable merge-on-write by default (#27188) 2023-12-06 21:06:58 +08:00
Pxl
5168422da7 [Chore](case) remove dynamic partition on test_delete case #28075 
remove dynamic partition on test_delete case to reduce logout
2023-12-06 16:23:12,733 WARN (DynamicPartitionScheduler|58) [DynamicPartitionScheduler.recordCreatePartitionFailedMsg():581] dynamic add partition failed: errCode = 2, detailMessage = Range [types: [DATETIMEV2]; keys: [0000-01-01 00:00:00]; ..types: [DATETIMEV2]; keys: [2021-01-01 00:00:00]; ) is intersected with range: [types: [DATETIMEV2]; keys: [2020-03-01 00:00:00]; ..types: [DATETIMEV2]; keys: [2020-04-01 00:00:00]; ), db: default_cluster:regression_test_delete_p0, table: dwd_pay
 2023-12-06 20:35:08 +08:00
Jerry Hu
b821672f8b [test](regression) add 'sync' for som stream load (#27357) 2023-11-22 10:52:34 +08:00
lsy3993
a510b5be81 [regression](delete) add regression test for every type delete (#26954) 2023-11-16 08:03:31 -06:00
Pxl
2712bb9f60 [Bug](decimalv2) getCmpType return decimalv2 when lhs/rhs type both is decimalv2 (#26705) 
getCmpType return decimalv2 when lhs/rhs type both is decimalv2
 2023-11-10 16:21:28 +08:00
zhangstar333
8f15f9adf6 [test](case) add test case to improve code coverage (#25516) 
[test](case) add test case to improve code coverage (#25516)
 2023-11-01 12:51:12 +08:00
amory
53b46b7e6c [FIX](filter) update for filter_by_select logic (#25007) 
this pr is aim to update for filter_by_select logic and change delete limit

only support scala type in delete statement where condition
only support column nullable and predict column support filter_by_select logic, because we can not push down non-scala type to storage layer to pack in predict column but do filter logic
 2023-10-09 21:27:40 +08:00
Pxl
ac6028a731 [Bug](partition) fix cannot delete from partitions (#24053) 2023-09-08 10:11:30 +08:00
mch_ucchi
db5a622166 [Fix](planner) fix incorrect nullable in ctas. (#22770) 
ctas with outer join like
```sql
create table a (
        id int not null,
        name varchar(20) not null
)
distributed by hash(id) buckets 4
properties (
        "replication_num"="1"
);

create table b (
        id int not null,
        age int not null
)
distributed by hash(id) buckets 4
properties (
        "replication_num"="1"
);

insert into a values(1, 'ww'), (2, 'zs');
insert into b values(1, 22);

create table c properties("replication_num"="1") as select a.id, a.name, b.age from a left join b on a.id = b.id;
```
the column 'age' in c is not null, but nullable is expected, we fix it by use the nullable mode of the outputs of root plan fragment.
 2023-08-21 11:46:27 +08:00
bobhan1
4510e16845 [improvement](delete) support delete predicate on value column for merge-on-write unique table (#21933) 
Previously, delete statement with conditions on value columns are only supported on duplicate tables. After we introduce delete sign mechanism to do batch delete, a delete statement with conditions on value columns on unique tables will be transformed into the corresponding insert into ..., __DELETE_SIGN__ select ... statement. However, for unique table with merge-on-write enabled, the overhead of inserting these data can be eliminated. So this PR add the ability to allow delete predicate on value columns for merge-on-write unique tables.
 2023-08-16 12:18:05 +08:00
mch_ucchi
36cea89c22 [Fix](planner)support delete conditions contain non-key columns and add check in analyze phase for delete. (#22673) 2023-08-07 21:49:53 +08:00
bobhan1
d9c93aaa1c [fix](regression) fix failed test delete_p0 in branch-2.0 #22652 2023-08-07 16:42:19 +08:00
yiguolei
75da60f1cc [regression-test](delete) should drop table test before create it (#22633) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2023-08-05 21:07:46 +08:00
mch_ucchi
3447a70b25 [Fix](planner)fix delete stmt contains where but delete all data. (#22563) 2023-08-03 23:44:05 +08:00
谢健
716d58f5ff [fix](Nereids) decimal divide should not return null if numerator is zero (#22309) 2023-07-27 20:23:04 +08:00
amory
3d0f952934 [FIX](complex-type)delete enable_map/struct_type switch #21957 2023-07-22 15:29:32 +08:00
zhangstar333
87556b5741 [bug](test) fix regression test case failed with curdate (#21922) 
fix regression test case failed with curdate
 2023-07-18 19:10:55 +08:00
Pxl
417e3e5616 [Feature](delete) support fold constant on delete stmt (#21833) 
support fold constant on delete stmt
 2023-07-18 12:56:28 +08:00
mch_ucchi
64a1eb77f0 [opt](planner) support delete with a subquery in predicate by construct an insert. (#20983) 
complex predicate in delete stmt like: 
```sql
delete from t1 where t1.id in (select id from t2);
```

will be replaced to an insert stmt.
```sql
insert into t1(id, __DORIS_DELETE_SIGN__) select id, 1 from t1 where id in (select id from t2);
```
 2023-06-27 17:51:13 +08:00
Jerry Hu
e90b78d783 [chore](regression) add case in test_delete (#20372) 
Add some cases of deletion conditions with numeric values.
 2023-06-05 09:38:29 +08:00
Jerry Hu
8ff8705b3f [fix](olap) deletion statement with space conditions did not take effect (#20349) 
Deletion statement like this:

delete from tb where k1 = '  ';
The rows whose k1's value is ' ' will not be deleted.
 2023-06-02 13:52:57 +08:00
amory
1ccbdee757 [FIX](map-type)fix map regress test & create mapTypeInfo without delete #19033 2023-04-26 19:03:55 +08:00
HappenLee
c993964a88 [Bug](delete) fix the delete ignore char case (#18714) 2023-04-26 07:30:44 +08:00
xueweizhang
798d2e5160 [fix](catalog) all properties should be checked when create unpartitioned table (#18149) 
all properties should be checked when create unpartitioned table like partitioned table.



Signed-off-by: nextdreamblue <zxw520blue1@163.com>
 2023-04-04 08:53:45 +08:00
HappenLee
473f0c45ff [Bug](delete) Fix bug of delete partition prune error (#18057) 2023-03-24 20:22:12 +08:00
morrySnow
fadf3b906d [enhancement](planner) delete support between predicate (#17892) 2023-03-23 13:24:32 +08:00
morrySnow
173d68409c [enhencement](planner) update and delete support use alias for target table (#17914) 2023-03-22 11:07:39 +08:00
morrySnow
0086fdbbdb [enhancement](planner) support delete from using syntax (#17787) 
support syntax delete using, this syntax only support UNIQUE KEY model

use the result of `t2` join `t3` to romve rows from `t1`

```sql
-- create t1, t2, t3 tables
CREATE TABLE t1
  (id INT, c1 BIGINT, c2 STRING, c3 DOUBLE, c4 DATE)
UNIQUE KEY (id)
DISTRIBUTED BY HASH (id)
PROPERTIES('replication_num'='1', "function_column.sequence_col" = "c4");

CREATE TABLE t2
  (id INT, c1 BIGINT, c2 STRING, c3 DOUBLE, c4 DATE)
DISTRIBUTED BY HASH (id)
PROPERTIES('replication_num'='1');

CREATE TABLE t3
  (id INT)
DISTRIBUTED BY HASH (id)
PROPERTIES('replication_num'='1');

-- insert data
INSERT INTO t1 VALUES
  (1, 1, '1', 1.0, '2000-01-01'),
  (2, 2, '2', 2.0, '2000-01-02'),
  (3, 3, '3', 3.0, '2000-01-03');

INSERT INTO t2 VALUES
  (1, 10, '10', 10.0, '2000-01-10'),
  (2, 20, '20', 20.0, '2000-01-20'),
  (3, 30, '30', 30.0, '2000-01-30'),
  (4, 4, '4', 4.0, '2000-01-04'),
  (5, 5, '5', 5.0, '2000-01-05');

INSERT INTO t3 VALUES
  (1),
  (4),
  (5);

-- remove rows from t1
DELETE FROM t1
  USING t2 INNER JOIN t3 ON t2.id = t3.id
  WHERE t1.id = t2.id;
```

the expect result is only remove the row where id = 1 in table t1

```
+----+----+----+--------+------------+
| id | c1 | c2 | c3     | c4         |
+----+----+----+--------+------------+
| 2  | 2  | 2  |    2.0 | 2000-01-02 |
| 3  | 3  | 3  |    3.0 | 2000-01-03 |
+----+----+----+--------+------------+
```
 2023-03-16 13:12:00 +08:00
xueweizhang
9155d8b9d1 [fix](delete) fix 'is null' or 'is not null' delete predicate will get wrong result (#17190) 
fix 'is null' or 'is not null' delete predicate will get wrong result

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
 2023-03-02 14:05:44 +08:00
xueweizhang
e0cd8599d2 [fix](delete) fix delete from bug which can get wrong result (#17146) 
理论上，如果是两次独立的删除，比如delete from table where a=1; delete from table where a=2;其实这个地方应该可以使用的，但是目前的代码，是把所有不同版本的delete predicates和不同列的delete predicates都放到一起了，失去了版本信息、失去了谓词间可能是and的关系，统一弱化成了delete predicates都是独立的，有一个delete predicates满足条件，就把page都去掉。
这个pr的修改方式，就是在当前代码的基础上，当只有一个delete predicate的时候才能保证后续淘汰page的正确性，所以这里一律加了 == 1的判断才传递delete predicates。
如果要把不同版本的delete predicates和不同列的delete predicates作为完整和严谨的逻辑去判断page，需要修改的设计就有点多了，目前的方案算是一种优先解决bug的思路，后续可以进一步把delete predicates这块加速zone判断进行page淘汰的逻辑完善，提高delete predicates使用的场景。
 2023-02-28 09:20:10 +08:00
xy720
91fc9fae8e [Bug](complex-type) Fix is null predicate in delete stmt for array/struct/map type (#17018) 2023-02-23 15:06:49 +08:00
xy720
6294b29f0a [chore](regression-test) Remove array config in regression test (#16376) 
The fe config "enable_array_type" is not used, this commit removes it from regression test.
 2023-02-03 14:44:03 +08:00
Gabriel
4dbe30d37b [regression](vectorized) delete vectorized config in regression tests (#15126) 2022-12-16 17:08:29 +08:00
Gabriel
5ef4c42a80 [Bug](datev2) Fix wrong result when use datev2 as partition key (#15094) 2022-12-15 21:27:05 +08:00