Commit Graph

12964 Commits

Author SHA1 Message Date
a136836770 [feature](Nereids) add two functions: char and covert (#23104)
add [char](https://doris.apache.org/zh-CN/docs/dev/sql-manual/sql-functions/string-functions/char/?_highlight=char) func
```
mysql> select char(68, 111, 114, 105, 115);
+--------------------------------------+
| char('utf8', 68, 111, 114, 105, 115) |
+--------------------------------------+
| Doris                                |
+--------------------------------------+
```

convert func
```
MySQL root@127.0.0.1:(none)> select convert(1 using gbk);
+-------------------+
| convert(1, 'gbk') |
+-------------------+
| 1                 |
+-------------------+
```
2023-08-30 17:09:06 +08:00
509d865760 [feature](Nereids): convert CaseWhen to If (#23040)
Add a rule to optimize CASE WHEN expression.
Rewrite rule to convert CASE WHEN to IF.

For example:
CASE WHEN a > 1 THEN 1 ELSE 0 END -> IF(a > 1, 1, 0)
2023-08-30 15:47:29 +08:00
3a0a79b4a0 [Improvement][SparkLoad] Use system env configs when users don't set env configs. (#21837) 2023-08-30 15:14:40 +08:00
1ce783fb23 [fix](stacktrace) Temporary fix ARM and MacOS stacktrace #23650 2023-08-30 14:51:20 +08:00
aef162ad4c [test](log) add some log in udf function when thrown exception (#23651)
[test](log) add some log in udf function when thrown exception (#23651)
2023-08-30 14:16:05 +08:00
4fec0826f8 [fix](Nereids): avoid Exception to cause analyze time too long (#23627)
AnyDataType will cause toCatalogDataType throw Exception, it will cost much time.

Avoid to throw Exception in Analyzer.
2023-08-30 12:25:31 +08:00
d326cb0c99 [fix](planner) array constructor do type coercion with decimal in wrong way (#23630)
array creator with decimal type and integer type parameters should return array<decimal>,
but the legacy planner return array<double>
2023-08-30 11:18:31 +08:00
942a119881 [bug](java-udf) fix java-udf not return const column when all args are const values (#23188)
eg: udf('asd'), this need return a const column, otherwise will be failed use the return column as other function params.

mysql> select concat('a', 'b', cuuid9('a'), ':c');
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.6)[CANCELLED][INTERNAL_ERROR]const check failed, expr=VectorizedFn[VectorizedFnCallconcat]{
VLiteral (name = String, type = String, value = (a)),
VLiteral (name = String, type = String, value = (b)),
VectorizedFn[VectorizedFnCallcuuid9]
{ VLiteral (name = String, type = String, value = (a))},
VLiteral (name = String, type = String, value = (:c))}
2023-08-30 10:46:47 +08:00
242e11aba1 [fix](case) disable case test_leading (#23641) 2023-08-30 10:24:06 +08:00
2a31a1e29b [Improvement](test) compute be full coverage (#23618) 2023-08-30 10:21:32 +08:00
f786689044 [refactor](TableRowCountAction) Fine-tune sql execution code (#23541) 2023-08-30 10:11:30 +08:00
ca55bd88ad [Fix](Job)Fix the window time is not updated when no job is registered (#23628)
Fix resume job grammar definition is inconsistent
Show Job task Add execution results
JOB allows to define update operations
2023-08-30 09:48:21 +08:00
e05a0466f2 [improve](Status) Add new status codeKEY_NOT_FOUND and KEY_ALREADY_EXISTS for merge on write (#23619) 2023-08-30 08:50:07 +08:00
d1dbe7bfc8 [fix](reader) fix leak in Level1Iteartor (#23612)
_merge_next() and _normal_next() leak _cur_child when _cur_child->next()
returns failure.
2023-08-29 23:32:24 +08:00
030df6db35 [fix](odbc) fix odbc insert string data to sqlserve (#23364) 2023-08-29 21:47:50 +08:00
e02747e976 [feature](Nereids) support struct type (#23597)
1. support struct data type
2. add array / map / struct literal syntax
3. fix array union / intersect / except type coercion
4. fix explict cast data type check for array
5. fix bound function type coercion
2023-08-29 20:41:24 +08:00
4f7e7040ad [bugfix] (dynamic partition) dynamic partition job is removed when tbl is sync (#23404) 2023-08-29 20:35:56 +08:00
1ac0ff0ea9 [feature](delete-predicate) support delete sub predicate v2 (#22442)
New structure for delete sub predicate.
Delete sub predicate uses a string type condition_str to stored temporarily now and fields will be extracted from it using std::regex, which may introduces stack overflow when matching a extremely large string(bug of libc).

Now we attempt to use a new PB structure to hold the delete sub predicate, to avoid that problem.

message DeleteSubPredicatePB {
    optional int32 column_unique_id = 1;
    optional string column_name = 2;
    optional string op = 3;
    optional string cond_value = 4;
}
Currently, 2 versions of sub predicate will both be filled. For query, we use the v2, and during compaction we still use v1. The old rowset meta with delete predicates which had sub predicate v1 will be attempted to convert to v2 when read from PB. Moreover, efforts will be made to rewrite these meta with the new delete sub predicate.

Make preparation to use column unique id to specify a column globally.
Using the column unique id rather than the column name to identify a column is vital for flexible schema change. The rewritten delete predicate will attach column unique id.
2023-08-29 19:37:23 +08:00
103fa4eb55 [feature](Export) support export with nereids (#23319) 2023-08-29 19:36:19 +08:00
94a8fa6bc9 [bug](function) fix explode_number function return wrong rows (#23603)
before the explode_number function result is random with const value.
because the _cur_size is reset, so it's can't insert values to column.
2023-08-29 19:02:49 +08:00
f17241386e [fix](regression) Fix test no_await #23599 2023-08-29 18:58:13 +08:00
cc1509ba11 [fix](view) The parameter positions of timestamp diff function to sql are reversed (#23601) 2023-08-29 18:30:16 +08:00
84006dd8c7 [Fix](Full compaction) Fix full compaction regressison test (#23487) 2023-08-29 18:27:19 +08:00
82a4f114e4 [improvement](compaction) add an option on delete stale rowset by judging _stale_rs_metas size when doing compaction (#23448) 2023-08-29 17:40:37 +08:00
1410a15a61 [fix](compaction) print column name when checking block ColumnPtr is nullptr on get block byte (#23338) 2023-08-29 17:24:48 +08:00
8932a6fae7 [feature](Nereids) support Literal collate syntax (#23600)
Support such sql grammar, Just for compatibility

```sql
select table_name
from information_schema.tables
where table_schema collate utf8_general_ci = 'information_schema'
  and table_name collate utf8_general_ci = 'parameters';
```
2023-08-29 17:01:13 +08:00
0cece561f9 [refactor](segment iterator) remove std::map in iterator use std::vector instead and not rely on unique id to idenfy position (#23505) 2023-08-29 16:43:32 +08:00
b372ded455 [disable](regression) disable test_stream_load_move_memtable (#23615) 2023-08-29 16:28:41 +08:00
f7a3d2778a [FIX](array)update array olapconvertor and support array nested other complex type (#23489)
* update array olapconvertor and support array nested other complex type

* update for inverted index
2023-08-29 16:18:11 +08:00
5fedb3285f [fix](regression) add sync for loading data to avoid case failure in multi-fe test env (#23604) 2023-08-29 15:29:51 +08:00
598dc6960a [fix](Nereids) make agg output unchanged after normalized (#23499)
The normalizedAgg rule can change the output of agg.

For example:
```
select c1 as c, c1 from t having c1 > 0
```
The normalizedAgg rule will make a plan with output c, which can cause the having filter error

Therefore, the output exprId should be unchanged after normalized
2023-08-29 15:01:26 +08:00
993659cd0b [FIX](serde) fix handle serde error #23565 2023-08-29 14:55:35 +08:00
4c00b1760b [feature](partial update) Support partial update for broker load (#22970) 2023-08-29 14:41:01 +08:00
97eb2b9172 [Fix](multi-catalog) Fix broker load reader and hdfs reader issue. (#23529)
Broker load with broker sometimes will throw 'Invalid orc post script length'.
hdfs query sometimes will throw 'Invalid orc post script length'.
2023-08-29 13:45:48 +08:00
7dcde4d529 [bug](decimal) Use max value as result if overflow (#23602)
* [bug](decimal) Use max value as result if overflow

* update
2023-08-29 13:26:25 +08:00
93db9b455a [test](fix case) fix sql user conflict in test case (#23583) 2023-08-29 11:33:49 +08:00
Pxl
7913354f78 add column number check for vsorted_run_merger (#23584) 2023-08-29 10:41:59 +08:00
fb67c53e5d [test](fix) remove unstable ds_rf95 test (#23593) 2023-08-29 10:35:58 +08:00
0128dd42d9 [fix](regexp_extract_all) fix be OOM when quering with regexp_extrac… (#23284) 2023-08-29 10:34:12 +08:00
5b641ebd40 [feature-wip](catalog) support deltalake catalog step1-metadata (#22493) 2023-08-29 10:31:37 +08:00
d8f159728b [fix](planner) only forbid substitute literal expr in function call expr (#23532)
This is a follow up pr of #23438. It's not correct to forbid substitute all literal exprs, only need to prevent substitute literal expr in function's param list.
2023-08-29 10:22:39 +08:00
1fd637eab7 [Docs](Job)Add Job and Job tasks docs (#23576) 2023-08-29 10:06:49 +08:00
da9eb79ac4 [Enhancement](Schema hash) Remove schema hash in tablet info (#23516) 2023-08-29 10:05:12 +08:00
d863cc3a12 [fix](move-memtable) fix tablets to commit (#23577) 2023-08-29 09:49:07 +08:00
b2cb0d99d4 [opt](docs) opt doc to avoid users using port 8030 everywhere (#23379) 2023-08-29 09:47:24 +08:00
9c65b7ab96 [improvement](column_reader) move load once to index reader to reduce (#23537)
memory footprint of column reader
2023-08-29 09:34:27 +08:00
fbf8499999 [improvement](compaction) reduce the memory using on vertical compaction (#23388) 2023-08-28 21:54:21 +08:00
6f3e2a30e6 [Feat](Nereids) Add leading and ordered hint (#22057)
Add leading hint and ordered hint. Usage:
select /*+ ordered / * from a join b on xxx; which will limit join order to original order
select /+ leading ({b a}) */ from a join b on xxx; which will change join order to b join a.
2023-08-28 21:04:40 +08:00
21fefb2831 [improvement](build) Compile enable WITH_MYSQL (#23371) 2023-08-28 20:41:06 +08:00
35a1404bbe [fix](load) add error handle when load data dir (#23457) 2023-08-28 19:33:50 +08:00