Commit Graph

781 Commits

Author SHA1 Message Date
81dbed70c2 [fix](Nereids) back off on tpch p1 (#16478)
adjust nullable on empty set should apply after unnested sub-query
some function should propagate nullable when args are datev2 or datetimev2
add back tpch sf0.1 nereids regression test
2023-02-08 10:43:13 +08:00
289a4b2ea4 [fix](func) fix truncate float type result error (#16468)
When the argument of truncate function is float type, it can match both truncate(DECIMALV3) and truncate(DOUBLE), if the match is truncate(DECIMALV3), the precision is lost when converting float to DECIMALV3(38, 0).

Here I modify it to match truncate(DOUBLE) for now, maybe we still need to solve the problem of losing precision when converting float to DECIMALV3.
2023-02-08 08:57:43 +08:00
1d0fdff98a [Bug](sort) disable 2phase read for sort by expressions exclude slotref (#16460)
```
create table tbl1 (k1 varchar(100), k2 string) distributed by hash(k1) buckets 1 properties("replication_num" = "1");

insert into tbl1 values(1, "alice");

select cast(k1 as INT) as id from tbl1 order by id limit 2;
```

The above query could pass `checkEnableTwoPhaseRead` since the order by element is SlotRef but actually it's an function call expr
2023-02-07 19:42:54 +08:00
91229bb87d [Bug](makr join) Fix mark join with other conjuncts (#16435) 2023-02-07 09:31:41 +08:00
a13beca0de [Fix](load)Use lower case for load column names. #16422
The columns name in stream load and broker load are case sensitive, make it case insensitive. This would be consist with query, because query sql columns name are case insensitve.
2023-02-07 09:18:37 +08:00
36a5e0a2a9 [bugfix](array) fix element revert on error in DataTypeArray::from_string (#16434)
* fix array from_string element revert on error

* add testcase
2023-02-06 18:27:36 +08:00
dccd04a3ba [fix](fe)predicate is wrongly pushed through CUBE function (#15831) 2023-02-06 11:29:15 +08:00
f2fd47f238 [Improve](row-store) support row cache (#16263) 2023-02-06 11:16:39 +08:00
f940cf4cf6 [fix](multi-catalog) fix recursive get schema cache bug (#16415) 2023-02-06 09:23:07 +08:00
09870098af [fix](func) fix core dump when the pattern of the regexp_extract_all function does not contain subpatterns (#16408) 2023-02-05 01:16:54 +08:00
df3a6e2412 [fix](fe)only set column info for slots in sortTupleDesc (#16407) 2023-02-04 23:14:25 +08:00
ca7b2e27a8 [regression-test](function) add regression test for money_format with truncate (#16052) 2023-02-04 23:10:01 +08:00
dd63897757 [fix](be)the set operation node should accept both nullable and non-nullable data from child node (#16126) 2023-02-04 23:08:59 +08:00
918004c016 [Bug](date) Fix BE crash caused by function datediff (#16397)
* [Bug](date) Fix BE crash caused by function `datediff`

* update
2023-02-04 18:43:23 +08:00
125b60b4b9 [improvement](compatibility) add DATA_TYPE in information schema for new types #16391
Add DATA_TYPE in information schema for types: datev2, datatimev2, decimal, jsonb. It was 'unknown' for these types and cause problem for tools such as BI using information schema.
2023-02-03 22:28:42 +08:00
4f778c38a1 [feature](nereids) support explore 4 phase aggregation (#16298)
support 4 phase Aggregation.
example: 
`select count(distinct k1), sum(k2) from t`
suppose t.k0 is distribute key.

we have plan 
```
Agg(DISTINCT_GLOBAL)
   |
Exchange(Gather)
  |
Agg(DISTINCT_LOCAL)
  |
Agg(GLOBAL)
  |
Exchange(hash distribute by k1)
 |
Agg(LOCAL) 
 |
scan
```

limitations:
1. only support sql with one distinct.
not support:`select count(distinct k1), count(distinct k2) from t`
2. only support sql with distinct one column
not support: `select count(distinct k1, k2) from t`
2023-02-03 21:51:10 +08:00
5e232a30d8 [fix](planner) Doris returns empty sets when select from a inline view (#16370)
Doris always delays the execution of expressions as possible as it can, so as the expansion of constant expression. Given below SQL:

```sql
select i from (select 'abc' as i, sum(birth) as j from  subquerytest2) as tmp
```

The aggregation would be eliminated, since its output is not required by the outer block, but the expasion for constant expression would be done in the final result expr, and since aggreagete output has been eliminate, the expasion would actually do nothing, and finally cause a empty results.

To fix this, we materialize the results expr in the inner block for such SQL, it may affect performance, but better than let system produce a mistaken result.
2023-02-03 21:23:52 +08:00
a5d9aca7ba [test](Nereids) enable G-K and L-Q scalar function regression test cases (#16169)
1. delete invalid signature of nvl function 
2. fix some test cases that failed because of malformed function name
2023-02-03 21:18:43 +08:00
87fbb8341a [Bug](datev2) Fix bug when cast datev2 to date (#16394) 2023-02-03 20:50:16 +08:00
929b31bd3c [Feature](Nereids) Support CaseWhen with subquery (#16385)
Co-authored-by: jianghaochen <jianghaochen@meituan.com>
2023-02-03 18:20:47 +08:00
3f4ca3da32 [Bug](CURRENT_TIMESTAMP) Fix wrong default value after schema change (#16364)
* [Bug](CURRENT_TIMESTAMP) Fix wrong default value after schema change

* update

* update
2023-02-03 17:06:24 +08:00
7d5a10e1af [bug](function) fix mask_first_n function can't handle const value (#16308) 2023-02-03 10:32:42 +08:00
Pxl
0d5b115993 [Feature](Materialized-View) support duplicate base column for diffrent aggregate function (#15837)
support duplicate base column for diffrent aggregate function
2023-02-02 18:57:39 +08:00
e31913faca [Feature](Nereids) Support order and limit in subquery (#15971)
1.Compatible with the old optimizer, the sort and limit in the subquery will not take effect, just delete it directly.
```
select * from sub_query_correlated_subquery1 where sub_query_correlated_subquery1.k1 > (select sum(sub_query_correlated_subquery3.k3) a from sub_query_correlated_subquery3 where sub_query_correlated_subquery3.v2 = sub_query_correlated_subquery1.k2 order by a limit 1);
```

2.Adjust the unnesting position of the subquery to ensure that the conjunct in the filter has been optimized, and then unnesting

Support:
```
SELECT DISTINCT k1 FROM sub_query_correlated_subquery1 i1 WHERE ((SELECT count(*) FROM sub_query_correlated_subquery1 WHERE ((k1 = i1.k1) AND (k2 = 2)) or ((k1 = i1.k1) AND (k2 = 1)) )  > 0);
```
The reason why the above can be supported is that conjunction will be performed, which can be converted into the following
```
SELECT DISTINCT k1 FROM sub_query_correlated_subquery1 i1 WHERE ((SELECT count(*) FROM sub_query_correlated_subquery1 WHERE ((k1 = i1.k1) AND (k2 = 2 or k2 = 1)) )  > 0);
```

Not Support:
```
SELECT DISTINCT k1 FROM sub_query_correlated_subquery1 i1 WHERE ((SELECT count(*) FROM sub_query_correlated_subquery1 WHERE ((k1 = i1.k1) AND (k2 = 2)) or ((k2 = i1.k1) AND (k2 = 1)) )  > 0);
```
2023-02-02 18:17:30 +08:00
557159d3ce [feature](JdbcExternalCatalog) support insert data in JdbcExternalCatalog (#16271) 2023-02-02 17:31:33 +08:00
09abd32957 [fix](test) result order in group-by-costant case is not stable (#16323) 2023-02-02 16:54:01 +08:00
68d2067f51 [improvement](testcase) change order by sql in test_dup_mv_bitmap_hash.groovy to make result stable
change order by sql in test_dup_mv_bitmap_hash.groovy to make result stable
2023-02-02 16:42:58 +08:00
bb179b77f7 [Feature-WIP](inverted index) support array type for inverted index reader (#16355) 2023-02-02 16:14:14 +08:00
69f34cd1c3 [fix](load) sequence column do not compare correctly in memtable (#16211) 2023-02-02 11:00:23 +08:00
1973b3a86f [test](regression) add tvf regression to test the remove of eof check (#16342)
Add regression test for #16302. This regression test will be failed if add EOF check for non-predicate columns.
2023-02-02 10:06:36 +08:00
941e192019 [enhancement](test) add function case date_sub(datetime,INTERVAL dayofmonth(datetime)-1 DAY) (#16306) 2023-02-02 09:56:01 +08:00
b878a7e61e [feature](Load)Suppot skip specific lines number for csv stream load (#16055)
Support set skip line number for stream load to load csv file.

Usage `-H skip_lines:number`:
```
curl --location-trusted -u root: -T test.csv -H skip_lines:5  -XPUT http://127.0.0.1:8030/api/testDb/testTbl/_stream_load
```

Skip line number also can be used in mysql load as below:
```sql
LOAD DATA
LOCAL
INFILE '${mysql_load_skip_lines}'
INTO TABLE ${tableName}
COLUMNS TERMINATED BY ','
IGNORE 2 LINES
PROPERTIES ("auth" = "root:");
```
2023-02-01 20:42:43 +08:00
bb0d4ba787 [BugFix](sort) use correct agg function when using 2 phase sort for agg table (#16185) 2023-02-01 20:07:43 +08:00
f8513dee2a [fix](Nereids): fix regression test in A-F.out (#16260) 2023-02-01 18:19:40 +08:00
e3c8fffd99 [function](round) fix decimal scale for scale not specified (#15541) 2023-02-01 14:58:48 +08:00
72a05a4358 [Bug](date) remove MinuteFloor/MinuteCeil for datev2 (#16247) 2023-02-01 14:57:51 +08:00
dbd1dfb64c [Bug](date) fix BE crash if month_floor 's argument is null (#16281) 2023-02-01 12:25:57 +08:00
95d7c2de26 [Refactor](function) Rewrite the function elt (#16287) 2023-02-01 11:17:06 +08:00
cd457312e4 [Enhancement](grouping) Add a switch for users to force using alias name in group by and having clause (#15748) 2023-01-31 23:46:31 +08:00
ca7eb94f23 [improvement](agg-function) Increase the limit maximum number of agg function parameters (#15924) 2023-01-31 21:03:50 +08:00
471db80f69 [Bug](date) Fix invalid date (#16205)
Issue Number: close #15777
2023-01-31 10:08:44 +08:00
e7c1d81419 [fix](planner) Pushdown constant predicate to all scan node in the lieteral view. #16217
Before this PR, planner might push a constant FALSE predicate to the wrong scan nodes in the literal view, and make this predicate useless
2023-01-30 22:18:43 +08:00
75c8670286 [Feature-WIP](inverted index) Filter out and remain predicates that do not support apply by inverted index, and add inverted index regression case (#16167)
1. Filter out and remain predicates that do not support applying on inverted index,
    like `BF` predicate, `IS_NULL` predicate, `IS_NOT_NULL` predicate.
2. Add inverted index regression case that based on tpcds_sf1 data set.
2023-01-30 22:16:08 +08:00
b480db2e11 [test](pipeline) Run nereids cases in p1/p2 (#16130) 2023-01-30 18:33:31 +08:00
342f3168b5 [fix](Nereids) return null when encryption is invalid rather than throw exception (#16234) 2023-01-30 16:52:42 +08:00
6bebf92254 [fix][FE] fix be coredump when children of FunctionCallExpr is folded (#16064)
Co-authored-by: shizhiqiang03 <shizhiqiang03@meituan.com>
fix be coredump when children of FunctionCallExpr is folded
2023-01-30 15:25:00 +08:00
5c00caa259 [refactor](Nereids) refactor BindSlotReference for easy merge all bind process in one rule (#16156) 2023-01-30 10:57:39 +08:00
bd1b7e190c [fix](Nereids): fix field(). (#16214) 2023-01-30 10:55:02 +08:00
69e748b076 [fix](schema scanner)change schema_scanner::get_next_row to get_next_block (#15718) 2023-01-30 10:01:50 +08:00
98649ec9f8 [fix](Nereids): Fix some functions error (#16197)
* fix bugs in regexp_extract_all

* fix rpad

* fix weekofday

* fix cryptor

* fix timestamp

* fix st_ function
2023-01-30 00:41:31 +08:00