Commit Graph

18429 Commits

Author SHA1 Message Date
6f47055f5a [opt](profile) Disable show query/load profile stmt (#32467) (#32813) 2024-03-26 13:56:46 +08:00
45245209ef [Serde](Variant) support arrow serialization for varint type (#32809) 2024-03-26 10:50:13 +08:00
6457a9a642 [fix](Nereids) system default decimalv3 scale should be 9 (#32754)
select round('1.1234', 2) should return 1.12, not 1
2024-03-26 10:43:49 +08:00
7b94cfdba1 Revert "[Fix](tests) add regression tests for trino-connector (#32552)"
This reverts commit 3fc3a4650681cb519405730899a2f22f268b38c1.
2024-03-25 22:38:21 +08:00
ff0da8108b [fix](RF) fix 'Invalid value' error of RF of decimal type (#32749) 2024-03-25 22:34:19 +08:00
3c74d4a4ef [fix](brpc) check failed socket before SetConnected (#32790) 2024-03-25 22:34:19 +08:00
41b9556a25 [pipelineX](localshuffle) Adjust local exchange strategy for distinct streaming agg (#32767)
Adjust local exchange strategy for distinct streaming agg
2024-03-25 22:33:36 +08:00
552bf5b41a [opt](jdbc catalog) close when jdbcClient is not empty (#32747) 2024-03-25 22:33:33 +08:00
de3b99be00 [fix](pipeline) fix check failed in StatefulOperator 2024-03-25 22:33:30 +08:00
0eb9256274 [Fix](TransientTask)Export tasks should only be run on the master node (#32700)
* [Fix](TransientTask)Export tasks should only be run on the master node
Add thread name

Export Task runs only on the master node, so it is necessary to explicitly start the corresponding resources. At the same time, refactor some code to avoid circular dependencies.

* TransientTaskManager is initialized twice. Therefore, the second initialization needs to be deleted.
2024-03-25 22:33:02 +08:00
0f5e13e17c [fix](Nereids) IGNORE_STORAGE_DATA_DISTRIBUTION should not block generating filter for nested loop join (#32653) 2024-03-25 22:33:02 +08:00
97a48ca6b7 [fix](mtmv)fix mysql MTMV do not automatically refresh even if the refresh method is COMPLETE (#32683) 2024-03-25 22:32:18 +08:00
3fc3a46506 [Fix](tests) add regression tests for trino-connector (#32552) 2024-03-25 22:31:55 +08:00
ae3542f552 [bugfix](use after free) should not set finish depdency any more if task ctx lock failed (#32730)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-03-25 22:31:55 +08:00
e71e1b6f30 [enhancement](threadname) change thread name more simple in order to see it in top -H (#31734)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-03-25 22:31:55 +08:00
b9ac5b3d5b [bugfix](hive)use originHiveKeys for hive partitionvalue (#32664) 2024-03-25 22:31:38 +08:00
bff45cf94f [fix](union) union source operator hold too many blocks from children (#32717) 2024-03-25 18:09:41 +08:00
e1968f9b64 [branch-2.1](tzdb) update Doris tzfiles to 2024a #32704 2024-03-24 21:53:08 +08:00
1b0cd4a4db [branch-2.1](routine-load) self-adaption backoff timeout (#32734)
* [opt](routine-load) self-adaption backoff timeout (#32227)

* [fix](routine-load) fix timeout backoff can not work (#32661)
2024-03-24 18:39:58 +08:00
55ae41000f [enhancement](fatal) change log fatal to throw exception to avoid core (#32715)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-03-24 12:08:57 +08:00
5c3ca0fbc2 [fix](move-memtable) fix load timeout caused by lost wakeup (#32720) 2024-03-24 08:07:01 +08:00
b9788e5e37 [fix] (Nereids) fix date function rewrite on datetimev1 column bug (#32569) 2024-03-24 08:07:01 +08:00
5c3fc818bb [Fix](Export) fix variable name of code #32637 2024-03-24 08:07:01 +08:00
5077f7dd9d [conf](mysql) opt mysql network timeout to 600s #32545 2024-03-24 08:07:01 +08:00
1d4e5a1c58 [enhance](auth)enable col auth (#32659) 2024-03-24 08:07:01 +08:00
33e40345b6 [Improve](build-script)Add be-extension-ignore to ignore avro-scanner when building release packages (#32699) 2024-03-24 08:06:46 +08:00
bd2e7d0ec8 [feature](hive)support insert overwrite (#32610)
support insert overwrite for unpartitioned table and partitioned table.

issue: #31442
2024-03-24 08:06:13 +08:00
ac8ba43d8d [fix](multi-catalog)resolve hive meta store compatibility for different version issues (#32551)
Fix hive list partition at different version.
Only Hive3 uses the prependCatalogToDbName() to wrap db_name.
2024-03-24 08:06:13 +08:00
f62cdecc08 [fix](Nereids) do not push down topn-filter through right/full outer join if the first orderkey is nulls first (#32633)
* do not push down topn-filter through right/full outer join if the first order key is nulls first
2024-03-24 08:06:13 +08:00
3d14d9e379 [fix](Nereids) fix bind having aggregate failed again (#32687)
follow up #32490

add more tests and fix some cases because some sqls are valid to mysql, but failed in doris
2024-03-24 08:06:13 +08:00
4ebbabf15e [test](fuzzy) test fuzzy in BE (#31607)
test fuzzy in BE
2024-03-24 08:06:13 +08:00
a55b5ea9ca [bug](udaf) fix memory leak in the java udaf (#32630)
fix memory leak in the java udaf
2024-03-24 08:06:13 +08:00
c223d9e7d0 [Fix](Test)Reduce the Sleep time and ensure that the corresponding thread resources are released after the case is executed. (#32665)
* [Fix](Test)Reduce the Sleep time and ensure that the corresponding thread resources are released after the case is executed.
* Fix load job id error
2024-03-24 08:06:13 +08:00
8b960beaec [refactor](nereids)unify outputTupleDesc and projection (#32093)
* unify join node output project
2024-03-24 08:05:42 +08:00
735f5213c1 Add log for case to help debug. (#32675) 2024-03-24 08:04:46 +08:00
8b34915518 [Fix](compress) Fix occasional crushes when serializing blocks (#32672) 2024-03-23 06:20:45 +08:00
35e580ec7a [fix](RF) fix 'Invalid value' error of RF of datetimev2 type for max value (#32649) 2024-03-22 22:29:50 +08:00
8514fabe16 Revert "[fix](routine-load) fix timeout backoff can not work (#32661)" (#32709)
This reverts commit 0d0f787d3e9901192a403d5eb61ea58c8ea17a8e.

Co-authored-by: stephen <hello-stephen@qq.com>
2024-03-22 22:27:37 +08:00
8af666b192 [branch-2.1](routine-load) enhance auto resume to keep routine load stable (#32689)
* enhance auto resume to keep routine load stable

* do not auto resume if job cannot resume definitely (#32419)
2024-03-22 18:07:12 +08:00
62c7d0a421 [Fix](point query) add query options for short circuit queries (#32530) (#32684)
Some options like `be_exec_version` needed for functions
2024-03-22 18:03:18 +08:00
38ce61da40 [Fix](variant) filter with variant access may lead to to parition/tablet prune fall through (#32560) (#32680)
Query like `select * from ut_p partitions(p2) where cast(var['a'] as int)  > 0` will fall through parition/tablet prunning since it's plan like
```
mysql> explain analyzed plan select * from ut_p where id = 3 and cast(var['a'] as int) = 789;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[26] ( outputExprs=[id#0, var#1] )                                                                                                                        |
| +--LogicalProject[25] ( distinct=false, projects=[id#0, var#1], excepts=[] )                                                                                               |
|    +--LogicalFilter[24] ( predicates=((cast(var#4 as INT) = 789) AND (id#0 = 3)) )                                                                                         |
|       +--LogicalFilter[23] ( predicates=(0 = __DORIS_DELETE_SIGN__#2) )                                                                                                    |
|          +--LogicalProject[22] ( distinct=false, projects=[id#0, var#1, __DORIS_DELETE_SIGN__#2, __DORIS_VERSION_COL__#3, element_at(var#1, 'a') AS `var`#4], excepts=[] ) |
|             +--LogicalOlapScan ( qualified=regression_test_variant_p0.ut_p, indexName=<index_not_selected>, selectedIndexId=10145, preAgg=ON )                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
```
with an extra LogicalProject on top of LogicalOlapScan, so we should handle such case to prune parition/tablet
2024-03-22 18:03:08 +08:00
fdc19b4892 [Fix](Variant) Initialize original_tablet_schema in _expand_variant_to_subcolumns to address potential nullptr issue (#32184) (#32678) 2024-03-22 18:02:58 +08:00
0d0f787d3e [fix](routine-load) fix timeout backoff can not work (#32661) 2024-03-22 16:38:52 +08:00
3f093627f2 [fix](metrics) fix compaction_used_permits are negative numbers (#32440)
Co-authored-by: hugoluo <hugoluo@tencent.com>
2024-03-22 16:38:52 +08:00
cb3a329169 [Update](test) update regression case for index change (#32656) 2024-03-22 16:38:21 +08:00
326a264fcd [Improvement](executor)Add spill property for workload group #32554 2024-03-22 16:38:19 +08:00
f443d6de85 [Fix](variant) filter with variant access may lead to to parition/tablet prune fall through (#32560)
Query like `select * from ut_p partitions(p2) where cast(var['a'] as int)  > 0` will fall through parition/tablet prunning since it's plan like
```
mysql> explain analyzed plan select * from ut_p where id = 3 and cast(var['a'] as int) = 789;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[26] ( outputExprs=[id#0, var#1] )                                                                                                                        |
| +--LogicalProject[25] ( distinct=false, projects=[id#0, var#1], excepts=[] )                                                                                               |
|    +--LogicalFilter[24] ( predicates=((cast(var#4 as INT) = 789) AND (id#0 = 3)) )                                                                                         |
|       +--LogicalFilter[23] ( predicates=(0 = __DORIS_DELETE_SIGN__#2) )                                                                                                    |
|          +--LogicalProject[22] ( distinct=false, projects=[id#0, var#1, __DORIS_DELETE_SIGN__#2, __DORIS_VERSION_COL__#3, element_at(var#1, 'a') AS `var`#4], excepts=[] ) |
|             +--LogicalOlapScan ( qualified=regression_test_variant_p0.ut_p, indexName=<index_not_selected>, selectedIndexId=10145, preAgg=ON )                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
```
with an extra LogicalProject on top of LogicalOlapScan, so we should handle such case to prune parition/tablet
2024-03-22 16:38:19 +08:00
e41311d77d [bug](fold) fix fold constant core dump with variant type (#32265)
1. variant type core dump at call get_data_at function, as not impl this function.
2. some case can't pass at old planner and fold_constant_by_be = on.
3. open enable_fold_constant_by_be = true.
2024-03-22 16:37:33 +08:00
8a6fc79797 [fix](routine-load) avoid routine load pause for check transaction status fail (#32638) 2024-03-22 16:36:49 +08:00
d30ad2ba04 [test](regression) delete invalid test cases (#32654) 2024-03-22 16:36:49 +08:00