Commit Graph

17749 Commits

Author SHA1 Message Date
4ebbabf15e [test](fuzzy) test fuzzy in BE (#31607)
test fuzzy in BE
2024-03-24 08:06:13 +08:00
a55b5ea9ca [bug](udaf) fix memory leak in the java udaf (#32630)
fix memory leak in the java udaf
2024-03-24 08:06:13 +08:00
c223d9e7d0 [Fix](Test)Reduce the Sleep time and ensure that the corresponding thread resources are released after the case is executed. (#32665)
* [Fix](Test)Reduce the Sleep time and ensure that the corresponding thread resources are released after the case is executed.
* Fix load job id error
2024-03-24 08:06:13 +08:00
8b960beaec [refactor](nereids)unify outputTupleDesc and projection (#32093)
* unify join node output project
2024-03-24 08:05:42 +08:00
735f5213c1 Add log for case to help debug. (#32675) 2024-03-24 08:04:46 +08:00
8b34915518 [Fix](compress) Fix occasional crushes when serializing blocks (#32672) 2024-03-23 06:20:45 +08:00
35e580ec7a [fix](RF) fix 'Invalid value' error of RF of datetimev2 type for max value (#32649) 2024-03-22 22:29:50 +08:00
8514fabe16 Revert "[fix](routine-load) fix timeout backoff can not work (#32661)" (#32709)
This reverts commit 0d0f787d3e9901192a403d5eb61ea58c8ea17a8e.

Co-authored-by: stephen <hello-stephen@qq.com>
2024-03-22 22:27:37 +08:00
8af666b192 [branch-2.1](routine-load) enhance auto resume to keep routine load stable (#32689)
* enhance auto resume to keep routine load stable

* do not auto resume if job cannot resume definitely (#32419)
2024-03-22 18:07:12 +08:00
62c7d0a421 [Fix](point query) add query options for short circuit queries (#32530) (#32684)
Some options like `be_exec_version` needed for functions
2024-03-22 18:03:18 +08:00
38ce61da40 [Fix](variant) filter with variant access may lead to to parition/tablet prune fall through (#32560) (#32680)
Query like `select * from ut_p partitions(p2) where cast(var['a'] as int)  > 0` will fall through parition/tablet prunning since it's plan like
```
mysql> explain analyzed plan select * from ut_p where id = 3 and cast(var['a'] as int) = 789;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[26] ( outputExprs=[id#0, var#1] )                                                                                                                        |
| +--LogicalProject[25] ( distinct=false, projects=[id#0, var#1], excepts=[] )                                                                                               |
|    +--LogicalFilter[24] ( predicates=((cast(var#4 as INT) = 789) AND (id#0 = 3)) )                                                                                         |
|       +--LogicalFilter[23] ( predicates=(0 = __DORIS_DELETE_SIGN__#2) )                                                                                                    |
|          +--LogicalProject[22] ( distinct=false, projects=[id#0, var#1, __DORIS_DELETE_SIGN__#2, __DORIS_VERSION_COL__#3, element_at(var#1, 'a') AS `var`#4], excepts=[] ) |
|             +--LogicalOlapScan ( qualified=regression_test_variant_p0.ut_p, indexName=<index_not_selected>, selectedIndexId=10145, preAgg=ON )                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
```
with an extra LogicalProject on top of LogicalOlapScan, so we should handle such case to prune parition/tablet
2024-03-22 18:03:08 +08:00
fdc19b4892 [Fix](Variant) Initialize original_tablet_schema in _expand_variant_to_subcolumns to address potential nullptr issue (#32184) (#32678) 2024-03-22 18:02:58 +08:00
0d0f787d3e [fix](routine-load) fix timeout backoff can not work (#32661) 2024-03-22 16:38:52 +08:00
3f093627f2 [fix](metrics) fix compaction_used_permits are negative numbers (#32440)
Co-authored-by: hugoluo <hugoluo@tencent.com>
2024-03-22 16:38:52 +08:00
cb3a329169 [Update](test) update regression case for index change (#32656) 2024-03-22 16:38:21 +08:00
326a264fcd [Improvement](executor)Add spill property for workload group #32554 2024-03-22 16:38:19 +08:00
f443d6de85 [Fix](variant) filter with variant access may lead to to parition/tablet prune fall through (#32560)
Query like `select * from ut_p partitions(p2) where cast(var['a'] as int)  > 0` will fall through parition/tablet prunning since it's plan like
```
mysql> explain analyzed plan select * from ut_p where id = 3 and cast(var['a'] as int) = 789;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[26] ( outputExprs=[id#0, var#1] )                                                                                                                        |
| +--LogicalProject[25] ( distinct=false, projects=[id#0, var#1], excepts=[] )                                                                                               |
|    +--LogicalFilter[24] ( predicates=((cast(var#4 as INT) = 789) AND (id#0 = 3)) )                                                                                         |
|       +--LogicalFilter[23] ( predicates=(0 = __DORIS_DELETE_SIGN__#2) )                                                                                                    |
|          +--LogicalProject[22] ( distinct=false, projects=[id#0, var#1, __DORIS_DELETE_SIGN__#2, __DORIS_VERSION_COL__#3, element_at(var#1, 'a') AS `var`#4], excepts=[] ) |
|             +--LogicalOlapScan ( qualified=regression_test_variant_p0.ut_p, indexName=<index_not_selected>, selectedIndexId=10145, preAgg=ON )                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
```
with an extra LogicalProject on top of LogicalOlapScan, so we should handle such case to prune parition/tablet
2024-03-22 16:38:19 +08:00
e41311d77d [bug](fold) fix fold constant core dump with variant type (#32265)
1. variant type core dump at call get_data_at function, as not impl this function.
2. some case can't pass at old planner and fold_constant_by_be = on.
3. open enable_fold_constant_by_be = true.
2024-03-22 16:37:33 +08:00
8a6fc79797 [fix](routine-load) avoid routine load pause for check transaction status fail (#32638) 2024-03-22 16:36:49 +08:00
d30ad2ba04 [test](regression) delete invalid test cases (#32654) 2024-03-22 16:36:49 +08:00
6812b575b2 [fix](Nereids) fix bind having aggregate failed (#32490)
fix bind having aggregate failed, keep the behavior like mysql
2024-03-22 16:36:46 +08:00
1c521cd94e [fix](backup) clear snapshotInfos and backupMeta when cancel (#32646) 2024-03-22 16:36:46 +08:00
a10466598b [fix](jdbc catalog) Fix query errors without jdbc pool default value on only BE upgrade (#32618) 2024-03-22 16:36:22 +08:00
2f2d488668 [opt](parquet) Support hive struct schema change (#32438)
Followup: #31128
This optimization allows doris to correctly read struct type data after changing the schema from hive.

## Changing  struct schema  in hive:
```sql
hive> create table struct_test(id int,sf struct<f1: int, f2: string>) stored as parquet;

hive> insert into struct_test values
    >           (1, named_struct('f1', 1, 'f2', 's1')),
    >           (2, named_struct('f1', 2, 'f2', 's2')),
    >           (3, named_struct('f1', 3, 'f2', 's3'));

hive> alter table struct_test change sf sf struct<f1:int, f3:string>;

hive> select * from struct_test;
OK
1	{"f1":1,"f3":null}
2	{"f1":2,"f3":null}
3	{"f1":3,"f3":null}
Time taken: 5.298 seconds, Fetched: 3 row(s)
```

The previous result of doris was:
```sql
mysql> select * from struct_test;
+------+-----------------------+
| id   | sf                    |
+------+-----------------------+
|    1 | {"f1": 1, "f3": "s1"} |
|    2 | {"f1": 2, "f3": "s2"} |
|    3 | {"f1": 3, "f3": "s3"} |
+------+-----------------------+
```

Now the result is same as hive:

```sql
mysql> select * from struct_test;
+------+-----------------------+
| id   | sf                    |
+------+-----------------------+
|    1 | {"f1": 1, "f3": null} |
|    2 | {"f1": 2, "f3": null} |
|    3 | {"f1": 3, "f3": null} |
+------+-----------------------+
```
2024-03-22 16:35:47 +08:00
01a5413e45 [fix](Nereids) filter-limit-project translate to wrong plan (#32496) 2024-03-22 16:35:47 +08:00
b2db5a4ded [fix](restore) fix restore odbc resource bug (#31989)
if some odbc tables use the same resource, and restore not all odbc tables, it will not retain the resource.
and check some conf for backup/restore

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2024-03-22 16:35:47 +08:00
4de8775e17 [feat](Nereids): rewrite sum literal to sum and count (#32244)
sum(v + 2) => sum(v) + 2*count(v)
sum(v - 2) => sum(v) - 2*count(v)
2024-03-22 16:35:47 +08:00
8f3f9a53be [feat](Nereids): add is null predicate for the first partition when updating mv by partition (#32463) 2024-03-22 16:35:47 +08:00
3f36aa2d48 [chore](Nereids) remove ensure project on top join (#32562) 2024-03-22 16:35:47 +08:00
4b34ecefbe [fix](nereids)str_to_date function's signature for folding constant is wrong (#32474) 2024-03-22 16:35:47 +08:00
647a0606aa [pipelineX](refactor) Wait for 2-phase execution before opening (#32613)
Wait for 2-phase execution before opening
2024-03-22 16:35:47 +08:00
39382a9774 [fix](Nereids): just pull up alias project above join through topn (#32305) 2024-03-22 16:35:47 +08:00
66336e59e6 [fix](join) the result of left semi join with empty right side should be false, not null (#32477) 2024-03-22 16:35:43 +08:00
baf3ae1a93 [refactor](nereids)unify outputTupleDesc and projection be part (#32439) 2024-03-22 16:35:43 +08:00
ab467f53db [fix](partition) Fix be tablet partition id eq 0 By report tablet (#32179) (#32667) 2024-03-22 15:38:58 +08:00
0f343e914a Revert "[case](Cloud) Add ssb case for hdfs vault (#32567)"
This reverts commit 60a673979e5b2f98e9cc66c77fa0d6c61b8ed0f7.
2024-03-22 15:26:09 +08:00
ebfb3418f9 [test](mtmv)Add tpch test cherry pick to branch 21 (#32611)
* [test](neredis) Add tpch test for query rewrite by materialized view (#30870)

query rewrite by materialized view sql is as following
q1, q5, q6, q8, q9, q12, q14
the other is not supported now, will be supported later

* change code usage
2024-03-22 15:20:38 +08:00
ea71472d64 [fix](build index) fix core when build index for a new column which without data (#32550) (#32669)
Co-authored-by: Luennng <luennng@gmail.com>
Co-authored-by: Tanya-W <tanya1218w@163,com>
2024-03-22 15:05:19 +08:00
a4a191fe56 [fix](index compaction)Fix MOW index compaction core (#32121) (#32657) 2024-03-22 14:20:19 +08:00
2e25b0079b fix compile 2024-03-22 09:17:40 +08:00
23c12fd68f [fix](join) core caused by null-safe-equal join (#32623) 2024-03-22 08:53:47 +08:00
921fab2196 [fix](memory) Fix thread context not initialized in MacOS (#32570) 2024-03-22 08:53:47 +08:00
e5d7ff64aa [fix](backup) clear snapshot info for cancelled backup job to reduce log size (#32604) 2024-03-22 08:53:17 +08:00
6b54171778 [bugfix](deadlock) pipelinex map lock should only scope in map not about pipelinectx's cancel method (#32622)
both global lock in fragment mgr should only protect the map logic, could not use it to protect cancel method.
fragment ctx cancel method should be protected by a lock.
query ctx cancel --> pipelinex fragment cancel ---> query ctx cancel will dead lock.
2024-03-22 08:52:38 +08:00
Pxl
6462d913ca [Improvement](brpc) log error message when AutoReleaseClosure meet brpc error or response… (#32628)
log error message when AutoReleaseClosure meet brpc error or response with error status
2024-03-22 08:52:38 +08:00
d3bdda6071 [fix](partial update) fix data correctness risk when load delete sign data into a table with sequence col (#32574) 2024-03-22 08:52:38 +08:00
55b7f7f019 [fix](inverted index) skip read index column data only for DUP and MOW table (#32594) 2024-03-22 08:52:16 +08:00
26efde2925 [test](Nereids) refine group by eliminate regression test (#32543) 2024-03-22 08:52:16 +08:00
06026b2f46 [opt](paimon) add split num and partition num (#32597)
1. And selected partition number indicator in PaimonScanNode in explain result.
2. Add raw file split and total split indicator in PaimonScanNode in explain result.
3. Opt the error msg of table valued function, print the error msg first to avoid the long msg be truncated.
2024-03-22 08:52:16 +08:00
8d6561b32d [opt](jdbc catalog) set jdbc test_connection default true (#32546) 2024-03-22 08:52:16 +08:00