Commit Graph

18263 Commits

Author SHA1 Message Date
a364a24ac2 [Enhance](regression) add hive out file check (#25475)
add hive out file check
fix hive sql state with " ; "
2023-10-17 10:11:57 +08:00
ef7d8aa99a [fix](be)confix bug of converting outer join probe block to nullable (#25492)
_do_evaluate will add temp result column into original table block, so in order to only convert correct columns to be nullable, need call convert_block_to_null before _do_evaluate
2023-10-17 10:10:56 +08:00
85b8497624 [fix](Tvf) return empty set when tvf queries an empty file or an error uri (#25280)
### Before:
return errors when tvf queries an empty file or an error uri:
1. get parsed schema failed, empty csv file
2. Can not get first file, please check uri.

### Now:
we just return empty set when tvf queries an empty file or an error uri.
```sql
mysql> select * from s3( 
"uri" = "https://error_uri/exp_1.csv", 
"s3.access_key"= "xx", 
"s3.secret_key" = "yy", 
"format" = "csv") limit 10;

Empty set (1.29 sec)
```
2023-10-17 09:52:53 +08:00
cda8fb6b8b [fix](load) return Status when error in RowsetWriter::build (#25381) 2023-10-17 09:40:23 +08:00
a194a15442 [improvement](tablet schedule) colocate balance between all groups (#23543) 2023-10-17 09:33:52 +08:00
f1a5e393c7 [feature](insert) Support group commit insert use new syntax like insert into table_id(xxx) (#25484) 2023-10-17 09:23:09 +08:00
f75ee49cb4 [chore](fmt) Remove stringstream by fmt (#25474)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-10-16 21:31:54 +08:00
59ebbb351e [feature](merge-cloud) Enable write into cache when uploading file to s3 using s3 file writer (#24364) 2023-10-16 21:31:02 +08:00
fe1980d7f2 [docs](docs) Add release note 2.0.2 (#25375) 2023-10-16 20:38:45 +08:00
f9a80ecdab [improvement](sync version) fe sync version with be (#25236) 2023-10-16 20:34:25 +08:00
e9157a3dba [fix](path gc) fix data dir path gc (#25420) 2023-10-16 20:25:20 +08:00
eaf5febc97 [enhancement](cooldown) Improve cooldown logs (#25432) 2023-10-16 20:17:00 +08:00
bfc602f343 [compile](fix) fix ubsan compile error (#25473) 2023-10-16 20:11:17 +08:00
Pxl
72920fbd1d [Improvement](materialized-view) set job failed when toAgentTaskRequest meet error (#25358)
set job failed when toAgentTaskRequest meet error
2023-10-16 20:10:52 +08:00
f9df3bae61 [Enhancement](functions) change some nullable mode and clear some smooth upgrade (#25334) 2023-10-16 19:50:17 +08:00
7fd876f3a2 [fix](planner)should call SlotRef'smaterializeSrcExpr() method if the slotRef is materialized (#25467) 2023-10-16 19:42:12 +08:00
dd1c4f4218 [fix](regression) fix group commit stream load regression test (#25469) 2023-10-16 19:41:51 +08:00
e3d0e55794 [feature-wip] (Nereids) Support transforming trino dialect SQL to logical plan (#21855)
Support transforming trino dialect SQL to logical plan (#21854)

## Proposed changes

Issue Number: #21854 
Use io.trino.sql.tree.AstVisitor as vistor, visit coorresponding trino node and transform it to doris logical plan.

## Further comments

Here are some examples for function transforming as following:
**ascii('a')** function is in doris and **codepoint('a')** funtion in trino, they have the same feature and have the same method signature, so we can use [TrinoFnCallTransformer](3b37b76886/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/trino/TrinoFnCallTransformer.java) to handle them.

another example for ComplexTransformer as following:
**date_diff('second', TIMESTAMP '2020-12-25 22:00:00', TIMESTAMP '2020-12-25 21:00:00')"** fuction in trino
and **seconds_diff(2020-12-25 22:00:00, 2020-12-25 21:00:00)")** fuction in doris. They have different method signature, we cant not handle it by TrinoFnCallTransformer simply and we should handle it by individual complex transformer [DateDiffFnCallTransformer](3b37b76886/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/trino/DateDiffFnCallTransformer.java).
2023-10-16 05:10:55 -05:00
cf073ec8ce [runtimefilter](nerieds)support Non equal runtime filter for nested loop join #25193 2023-10-16 17:49:47 +08:00
Pxl
d41e839ea0 [Chore](sink) add index number check for table sink (#25461)
add index number check for table sink
2023-10-16 17:03:26 +08:00
9deda929b9 [refactor](stats) Use id instead name in analysis info (#25213) 2023-10-16 03:49:53 -05:00
4c42f3b783 [Improvement](hive-udf)(doc) minimize hive-udf and add some docs. (#24786) 2023-10-16 16:47:21 +08:00
5eff36417a [typo](docs) Fix some ambiguous descriptions (#23912) 2023-10-16 16:44:11 +08:00
b2e3ecb81d [opt](load)change load_to_single_tablet tablet search algorithm from random to round-robin (#25256)
At present, `load_to_singlt_tablet` import implementation refers to simple random number remainder, which cannot achieve true averaging. This will lead to uneven disk IO and uneven use of cluster resources. To solve this problem, we are preparing to implement round-robin for each partition tablet imported each time, in order to achieve average load to each tablet.

When generating the load query plan, the tablet index record currently imported is passed to BE.
Add a deamon task in FE to regularly clean up the `loadTabletRecordMap`. The map will get the bucket_number of the partition and update the `load_tablet_index` when `getCurrentLoadTabletIndex`.
2023-10-16 16:43:25 +08:00
b83e412623 [fix](hive-udf) delete Logger to avoid Kryo serialize problem. (#25312) 2023-10-16 16:10:06 +08:00
e8431e1a97 [fix](planner)should not add TupleIsNullPredicate for inlineview plan (#25338) 2023-10-16 15:24:13 +08:00
8e9e1b1bfd [fix](planner) Disable infer expr column name when query on old optimizer (#25317)
Disable infer expr column name when query on old optimizer.
This bug is be brought in #24990

if your query SQL is
select id, name, sum(target) FROM db_test.table_test2 group by id, name;
the result column name when query is as following:
|id|name |sum(cast(target as DOUBLE))|

when you create view as following:
CREATE VIEW v1 as select id, name, sum(target) FROM db_test.table_test2 group by id, name;
then query the view v1, the result is as following:
|id|name |__sum_2|
2023-10-16 02:08:52 -05:00
1a27ac8d56 [opt] use correct column label when execute query in FE (#25372)
SET @a = '4';
SELECT @a;

previous:
+-----+
| '4' |
+-----+
| 4   |
+-----+

current:
+----+
| @a |
+----+
| 4  |
+----+
2023-10-16 02:03:33 -05:00
Pxl
292ccaeda8 insert default when json array parse failed (#25447)
insert default when json array parse failed
2023-10-16 14:51:26 +08:00
04e5fb3809 [fix](regression test) fix mysql tuple convert test result not ordered #25455 2023-10-16 14:32:51 +08:00
0585beee02 [typo](docs) Modify parameter description (#23782) 2023-10-16 01:29:00 -05:00
0aa50fb256 [fix](nereids)fix regression case: eliminate_outer_join (#25208) 2023-10-16 14:08:36 +08:00
f698f205d5 [Fix](merge-on-write) throw exception when the user don't specify the insert columns in insert statement for partial update (#25437) 2023-10-16 14:05:06 +08:00
e94fbe169e [Enhance](regression) add hms catalog broker scan case (#25453) 2023-10-16 12:35:46 +08:00
29d4e8ee90 [Fix](Nereids) fix test leading change disable join reorder parameter (#23657)
Problem:
when running pipeline, we get randomly failed of test_leading
Reason:
physical distribute was generated and choosed to be the best plan because we can not get any statistic information of empty table. So we would get some unexpect result because we can not expect the order in memo
Solved:
Add statistic of columns used in test_leading, try repeatly in pipeline
2023-10-15 22:59:45 -05:00
Pxl
d00d029ffb Separate fixed key hash map context creator (#25438)
Separate fixed key hash map context creator
2023-10-16 11:20:30 +08:00
c482c22a74 [case](regresscases) add regress cases for nested type nested type with csv format (#25355)
this pr
1.  fix use podarray push_back() with back() will make heap_use_after_free when podarray is reach capacity which would may make heap free 
2. add cases for csv format for nested types. and csv file has two define which are without quote or just like json text
2023-10-16 11:13:44 +08:00
934d82816c [fix](Nereids) add int type alias 'integer' (#25376) 2023-10-15 22:12:44 -05:00
4c57c31c5c [fix](Nereids) count should not accept complex and json type (#25354) 2023-10-15 22:08:35 -05:00
dfc7d04626 [fix](functions) add quantile_state_empty function signature (#25306) 2023-10-16 11:05:48 +08:00
9649e09aaa [feature](function) support bitmap type in min/max_by agg function (#25430)
support bitmap type in min/max_by agg function
2023-10-16 11:05:32 +08:00
97c0af1a80 [fix](build) aarch64 compilation fix # (#25443)
Issue: #25442

Compilation to include execinfo when building on aarch64
2023-10-16 09:53:50 +08:00
08f305dd79 [chore](build) Fix compilation errors reported by GCC-13 (#25439)
1. Fix lots of compilation errors reported by GCC-13.
2. Fix the workflow BE UT (macOS).
2023-10-15 07:57:36 -05:00
7ea456ef91 [fix](insert) make group commit wal_manager exit elegantly (#25250) 2023-10-14 23:14:06 +08:00
471cf2c48b [improvement](auth) support show view priv (#25370)
Issue Number: close #xxx

current ,if user has select_priv or load_priv,he can show create table view_name,
but this is not safe,so add show_view_priv for show create table view_name

mysql SHOW VIEW description: https://dev.mysql.com/doc/refman/8.0/en/privileges-provided.html#priv_show-view
2023-10-14 22:37:51 +08:00
ed3e8f9355 Fix a naming conflict in ccr regression test (#25436) 2023-10-14 22:28:59 +08:00
03316e2355 [fix](fe rest api)api gets execution plan, table name case problem (#25112)
The user has configured the parameter lower_case_table_names, which ignores the case of the table name. When executed on the SQL client, the table name can be queried in both case.
But when using Connector to read doris data, the table names must be in the same case, otherwise an error will be reported.
2023-10-14 19:48:24 +08:00
e5ef0aa6d4 [refactor](mysql result format) use new serde framework to tuple convert (#25006) 2023-10-14 19:46:42 +08:00
b946521a56 [enhancement](regression-test) add single stream multi table case (#25360) 2023-10-14 10:59:50 +08:00
de03c152ce [fix](thrift)cancel thrift msg max size limit (#25194)
On Thrift 0.14.0+, need use TConfiguration to raise the max message size.
see https://github.com/apache/arrow/pull/11123/files
2023-10-13 20:21:36 +08:00