Commit Graph

18263 Commits

Author SHA1 Message Date
bff45cf94f [fix](union) union source operator hold too many blocks from children (#32717) 2024-03-25 18:09:41 +08:00
e1968f9b64 [branch-2.1](tzdb) update Doris tzfiles to 2024a #32704 2024-03-24 21:53:08 +08:00
1b0cd4a4db [branch-2.1](routine-load) self-adaption backoff timeout (#32734)
* [opt](routine-load) self-adaption backoff timeout (#32227)

* [fix](routine-load) fix timeout backoff can not work (#32661)
2024-03-24 18:39:58 +08:00
55ae41000f [enhancement](fatal) change log fatal to throw exception to avoid core (#32715)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-03-24 12:08:57 +08:00
5c3ca0fbc2 [fix](move-memtable) fix load timeout caused by lost wakeup (#32720) 2024-03-24 08:07:01 +08:00
b9788e5e37 [fix] (Nereids) fix date function rewrite on datetimev1 column bug (#32569) 2024-03-24 08:07:01 +08:00
5c3fc818bb [Fix](Export) fix variable name of code #32637 2024-03-24 08:07:01 +08:00
5077f7dd9d [conf](mysql) opt mysql network timeout to 600s #32545 2024-03-24 08:07:01 +08:00
1d4e5a1c58 [enhance](auth)enable col auth (#32659) 2024-03-24 08:07:01 +08:00
33e40345b6 [Improve](build-script)Add be-extension-ignore to ignore avro-scanner when building release packages (#32699) 2024-03-24 08:06:46 +08:00
bd2e7d0ec8 [feature](hive)support insert overwrite (#32610)
support insert overwrite for unpartitioned table and partitioned table.

issue: #31442
2024-03-24 08:06:13 +08:00
ac8ba43d8d [fix](multi-catalog)resolve hive meta store compatibility for different version issues (#32551)
Fix hive list partition at different version.
Only Hive3 uses the prependCatalogToDbName() to wrap db_name.
2024-03-24 08:06:13 +08:00
f62cdecc08 [fix](Nereids) do not push down topn-filter through right/full outer join if the first orderkey is nulls first (#32633)
* do not push down topn-filter through right/full outer join if the first order key is nulls first
2024-03-24 08:06:13 +08:00
3d14d9e379 [fix](Nereids) fix bind having aggregate failed again (#32687)
follow up #32490

add more tests and fix some cases because some sqls are valid to mysql, but failed in doris
2024-03-24 08:06:13 +08:00
4ebbabf15e [test](fuzzy) test fuzzy in BE (#31607)
test fuzzy in BE
2024-03-24 08:06:13 +08:00
a55b5ea9ca [bug](udaf) fix memory leak in the java udaf (#32630)
fix memory leak in the java udaf
2024-03-24 08:06:13 +08:00
c223d9e7d0 [Fix](Test)Reduce the Sleep time and ensure that the corresponding thread resources are released after the case is executed. (#32665)
* [Fix](Test)Reduce the Sleep time and ensure that the corresponding thread resources are released after the case is executed.
* Fix load job id error
2024-03-24 08:06:13 +08:00
8b960beaec [refactor](nereids)unify outputTupleDesc and projection (#32093)
* unify join node output project
2024-03-24 08:05:42 +08:00
735f5213c1 Add log for case to help debug. (#32675) 2024-03-24 08:04:46 +08:00
8b34915518 [Fix](compress) Fix occasional crushes when serializing blocks (#32672) 2024-03-23 06:20:45 +08:00
35e580ec7a [fix](RF) fix 'Invalid value' error of RF of datetimev2 type for max value (#32649) 2024-03-22 22:29:50 +08:00
8514fabe16 Revert "[fix](routine-load) fix timeout backoff can not work (#32661)" (#32709)
This reverts commit 0d0f787d3e9901192a403d5eb61ea58c8ea17a8e.

Co-authored-by: stephen <hello-stephen@qq.com>
2024-03-22 22:27:37 +08:00
8af666b192 [branch-2.1](routine-load) enhance auto resume to keep routine load stable (#32689)
* enhance auto resume to keep routine load stable

* do not auto resume if job cannot resume definitely (#32419)
2024-03-22 18:07:12 +08:00
62c7d0a421 [Fix](point query) add query options for short circuit queries (#32530) (#32684)
Some options like `be_exec_version` needed for functions
2024-03-22 18:03:18 +08:00
38ce61da40 [Fix](variant) filter with variant access may lead to to parition/tablet prune fall through (#32560) (#32680)
Query like `select * from ut_p partitions(p2) where cast(var['a'] as int)  > 0` will fall through parition/tablet prunning since it's plan like
```
mysql> explain analyzed plan select * from ut_p where id = 3 and cast(var['a'] as int) = 789;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[26] ( outputExprs=[id#0, var#1] )                                                                                                                        |
| +--LogicalProject[25] ( distinct=false, projects=[id#0, var#1], excepts=[] )                                                                                               |
|    +--LogicalFilter[24] ( predicates=((cast(var#4 as INT) = 789) AND (id#0 = 3)) )                                                                                         |
|       +--LogicalFilter[23] ( predicates=(0 = __DORIS_DELETE_SIGN__#2) )                                                                                                    |
|          +--LogicalProject[22] ( distinct=false, projects=[id#0, var#1, __DORIS_DELETE_SIGN__#2, __DORIS_VERSION_COL__#3, element_at(var#1, 'a') AS `var`#4], excepts=[] ) |
|             +--LogicalOlapScan ( qualified=regression_test_variant_p0.ut_p, indexName=<index_not_selected>, selectedIndexId=10145, preAgg=ON )                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
```
with an extra LogicalProject on top of LogicalOlapScan, so we should handle such case to prune parition/tablet
2024-03-22 18:03:08 +08:00
fdc19b4892 [Fix](Variant) Initialize original_tablet_schema in _expand_variant_to_subcolumns to address potential nullptr issue (#32184) (#32678) 2024-03-22 18:02:58 +08:00
0d0f787d3e [fix](routine-load) fix timeout backoff can not work (#32661) 2024-03-22 16:38:52 +08:00
3f093627f2 [fix](metrics) fix compaction_used_permits are negative numbers (#32440)
Co-authored-by: hugoluo <hugoluo@tencent.com>
2024-03-22 16:38:52 +08:00
cb3a329169 [Update](test) update regression case for index change (#32656) 2024-03-22 16:38:21 +08:00
326a264fcd [Improvement](executor)Add spill property for workload group #32554 2024-03-22 16:38:19 +08:00
f443d6de85 [Fix](variant) filter with variant access may lead to to parition/tablet prune fall through (#32560)
Query like `select * from ut_p partitions(p2) where cast(var['a'] as int)  > 0` will fall through parition/tablet prunning since it's plan like
```
mysql> explain analyzed plan select * from ut_p where id = 3 and cast(var['a'] as int) = 789;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[26] ( outputExprs=[id#0, var#1] )                                                                                                                        |
| +--LogicalProject[25] ( distinct=false, projects=[id#0, var#1], excepts=[] )                                                                                               |
|    +--LogicalFilter[24] ( predicates=((cast(var#4 as INT) = 789) AND (id#0 = 3)) )                                                                                         |
|       +--LogicalFilter[23] ( predicates=(0 = __DORIS_DELETE_SIGN__#2) )                                                                                                    |
|          +--LogicalProject[22] ( distinct=false, projects=[id#0, var#1, __DORIS_DELETE_SIGN__#2, __DORIS_VERSION_COL__#3, element_at(var#1, 'a') AS `var`#4], excepts=[] ) |
|             +--LogicalOlapScan ( qualified=regression_test_variant_p0.ut_p, indexName=<index_not_selected>, selectedIndexId=10145, preAgg=ON )                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
```
with an extra LogicalProject on top of LogicalOlapScan, so we should handle such case to prune parition/tablet
2024-03-22 16:38:19 +08:00
e41311d77d [bug](fold) fix fold constant core dump with variant type (#32265)
1. variant type core dump at call get_data_at function, as not impl this function.
2. some case can't pass at old planner and fold_constant_by_be = on.
3. open enable_fold_constant_by_be = true.
2024-03-22 16:37:33 +08:00
8a6fc79797 [fix](routine-load) avoid routine load pause for check transaction status fail (#32638) 2024-03-22 16:36:49 +08:00
d30ad2ba04 [test](regression) delete invalid test cases (#32654) 2024-03-22 16:36:49 +08:00
6812b575b2 [fix](Nereids) fix bind having aggregate failed (#32490)
fix bind having aggregate failed, keep the behavior like mysql
2024-03-22 16:36:46 +08:00
1c521cd94e [fix](backup) clear snapshotInfos and backupMeta when cancel (#32646) 2024-03-22 16:36:46 +08:00
a10466598b [fix](jdbc catalog) Fix query errors without jdbc pool default value on only BE upgrade (#32618) 2024-03-22 16:36:22 +08:00
2f2d488668 [opt](parquet) Support hive struct schema change (#32438)
Followup: #31128
This optimization allows doris to correctly read struct type data after changing the schema from hive.

## Changing  struct schema  in hive:
```sql
hive> create table struct_test(id int,sf struct<f1: int, f2: string>) stored as parquet;

hive> insert into struct_test values
    >           (1, named_struct('f1', 1, 'f2', 's1')),
    >           (2, named_struct('f1', 2, 'f2', 's2')),
    >           (3, named_struct('f1', 3, 'f2', 's3'));

hive> alter table struct_test change sf sf struct<f1:int, f3:string>;

hive> select * from struct_test;
OK
1	{"f1":1,"f3":null}
2	{"f1":2,"f3":null}
3	{"f1":3,"f3":null}
Time taken: 5.298 seconds, Fetched: 3 row(s)
```

The previous result of doris was:
```sql
mysql> select * from struct_test;
+------+-----------------------+
| id   | sf                    |
+------+-----------------------+
|    1 | {"f1": 1, "f3": "s1"} |
|    2 | {"f1": 2, "f3": "s2"} |
|    3 | {"f1": 3, "f3": "s3"} |
+------+-----------------------+
```

Now the result is same as hive:

```sql
mysql> select * from struct_test;
+------+-----------------------+
| id   | sf                    |
+------+-----------------------+
|    1 | {"f1": 1, "f3": null} |
|    2 | {"f1": 2, "f3": null} |
|    3 | {"f1": 3, "f3": null} |
+------+-----------------------+
```
2024-03-22 16:35:47 +08:00
01a5413e45 [fix](Nereids) filter-limit-project translate to wrong plan (#32496) 2024-03-22 16:35:47 +08:00
b2db5a4ded [fix](restore) fix restore odbc resource bug (#31989)
if some odbc tables use the same resource, and restore not all odbc tables, it will not retain the resource.
and check some conf for backup/restore

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2024-03-22 16:35:47 +08:00
4de8775e17 [feat](Nereids): rewrite sum literal to sum and count (#32244)
sum(v + 2) => sum(v) + 2*count(v)
sum(v - 2) => sum(v) - 2*count(v)
2024-03-22 16:35:47 +08:00
8f3f9a53be [feat](Nereids): add is null predicate for the first partition when updating mv by partition (#32463) 2024-03-22 16:35:47 +08:00
3f36aa2d48 [chore](Nereids) remove ensure project on top join (#32562) 2024-03-22 16:35:47 +08:00
4b34ecefbe [fix](nereids)str_to_date function's signature for folding constant is wrong (#32474) 2024-03-22 16:35:47 +08:00
647a0606aa [pipelineX](refactor) Wait for 2-phase execution before opening (#32613)
Wait for 2-phase execution before opening
2024-03-22 16:35:47 +08:00
39382a9774 [fix](Nereids): just pull up alias project above join through topn (#32305) 2024-03-22 16:35:47 +08:00
66336e59e6 [fix](join) the result of left semi join with empty right side should be false, not null (#32477) 2024-03-22 16:35:43 +08:00
baf3ae1a93 [refactor](nereids)unify outputTupleDesc and projection be part (#32439) 2024-03-22 16:35:43 +08:00
ab467f53db [fix](partition) Fix be tablet partition id eq 0 By report tablet (#32179) (#32667) 2024-03-22 15:38:58 +08:00
0f343e914a Revert "[case](Cloud) Add ssb case for hdfs vault (#32567)"
This reverts commit 60a673979e5b2f98e9cc66c77fa0d6c61b8ed0f7.
2024-03-22 15:26:09 +08:00