Commit Graph

8289 Commits

Author SHA1 Message Date
f62cdecc08 [fix](Nereids) do not push down topn-filter through right/full outer join if the first orderkey is nulls first (#32633)
* do not push down topn-filter through right/full outer join if the first order key is nulls first
2024-03-24 08:06:13 +08:00
3d14d9e379 [fix](Nereids) fix bind having aggregate failed again (#32687)
follow up #32490

add more tests and fix some cases because some sqls are valid to mysql, but failed in doris
2024-03-24 08:06:13 +08:00
c223d9e7d0 [Fix](Test)Reduce the Sleep time and ensure that the corresponding thread resources are released after the case is executed. (#32665)
* [Fix](Test)Reduce the Sleep time and ensure that the corresponding thread resources are released after the case is executed.
* Fix load job id error
2024-03-24 08:06:13 +08:00
8b960beaec [refactor](nereids)unify outputTupleDesc and projection (#32093)
* unify join node output project
2024-03-24 08:05:42 +08:00
8b34915518 [Fix](compress) Fix occasional crushes when serializing blocks (#32672) 2024-03-23 06:20:45 +08:00
8514fabe16 Revert "[fix](routine-load) fix timeout backoff can not work (#32661)" (#32709)
This reverts commit 0d0f787d3e9901192a403d5eb61ea58c8ea17a8e.

Co-authored-by: stephen <hello-stephen@qq.com>
2024-03-22 22:27:37 +08:00
8af666b192 [branch-2.1](routine-load) enhance auto resume to keep routine load stable (#32689)
* enhance auto resume to keep routine load stable

* do not auto resume if job cannot resume definitely (#32419)
2024-03-22 18:07:12 +08:00
62c7d0a421 [Fix](point query) add query options for short circuit queries (#32530) (#32684)
Some options like `be_exec_version` needed for functions
2024-03-22 18:03:18 +08:00
0d0f787d3e [fix](routine-load) fix timeout backoff can not work (#32661) 2024-03-22 16:38:52 +08:00
326a264fcd [Improvement](executor)Add spill property for workload group #32554 2024-03-22 16:38:19 +08:00
f443d6de85 [Fix](variant) filter with variant access may lead to to parition/tablet prune fall through (#32560)
Query like `select * from ut_p partitions(p2) where cast(var['a'] as int)  > 0` will fall through parition/tablet prunning since it's plan like
```
mysql> explain analyzed plan select * from ut_p where id = 3 and cast(var['a'] as int) = 789;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[26] ( outputExprs=[id#0, var#1] )                                                                                                                        |
| +--LogicalProject[25] ( distinct=false, projects=[id#0, var#1], excepts=[] )                                                                                               |
|    +--LogicalFilter[24] ( predicates=((cast(var#4 as INT) = 789) AND (id#0 = 3)) )                                                                                         |
|       +--LogicalFilter[23] ( predicates=(0 = __DORIS_DELETE_SIGN__#2) )                                                                                                    |
|          +--LogicalProject[22] ( distinct=false, projects=[id#0, var#1, __DORIS_DELETE_SIGN__#2, __DORIS_VERSION_COL__#3, element_at(var#1, 'a') AS `var`#4], excepts=[] ) |
|             +--LogicalOlapScan ( qualified=regression_test_variant_p0.ut_p, indexName=<index_not_selected>, selectedIndexId=10145, preAgg=ON )                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
```
with an extra LogicalProject on top of LogicalOlapScan, so we should handle such case to prune parition/tablet
2024-03-22 16:38:19 +08:00
e41311d77d [bug](fold) fix fold constant core dump with variant type (#32265)
1. variant type core dump at call get_data_at function, as not impl this function.
2. some case can't pass at old planner and fold_constant_by_be = on.
3. open enable_fold_constant_by_be = true.
2024-03-22 16:37:33 +08:00
8a6fc79797 [fix](routine-load) avoid routine load pause for check transaction status fail (#32638) 2024-03-22 16:36:49 +08:00
6812b575b2 [fix](Nereids) fix bind having aggregate failed (#32490)
fix bind having aggregate failed, keep the behavior like mysql
2024-03-22 16:36:46 +08:00
1c521cd94e [fix](backup) clear snapshotInfos and backupMeta when cancel (#32646) 2024-03-22 16:36:46 +08:00
a10466598b [fix](jdbc catalog) Fix query errors without jdbc pool default value on only BE upgrade (#32618) 2024-03-22 16:36:22 +08:00
01a5413e45 [fix](Nereids) filter-limit-project translate to wrong plan (#32496) 2024-03-22 16:35:47 +08:00
b2db5a4ded [fix](restore) fix restore odbc resource bug (#31989)
if some odbc tables use the same resource, and restore not all odbc tables, it will not retain the resource.
and check some conf for backup/restore

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2024-03-22 16:35:47 +08:00
4de8775e17 [feat](Nereids): rewrite sum literal to sum and count (#32244)
sum(v + 2) => sum(v) + 2*count(v)
sum(v - 2) => sum(v) - 2*count(v)
2024-03-22 16:35:47 +08:00
8f3f9a53be [feat](Nereids): add is null predicate for the first partition when updating mv by partition (#32463) 2024-03-22 16:35:47 +08:00
3f36aa2d48 [chore](Nereids) remove ensure project on top join (#32562) 2024-03-22 16:35:47 +08:00
4b34ecefbe [fix](nereids)str_to_date function's signature for folding constant is wrong (#32474) 2024-03-22 16:35:47 +08:00
39382a9774 [fix](Nereids): just pull up alias project above join through topn (#32305) 2024-03-22 16:35:47 +08:00
ab467f53db [fix](partition) Fix be tablet partition id eq 0 By report tablet (#32179) (#32667) 2024-03-22 15:38:58 +08:00
ebfb3418f9 [test](mtmv)Add tpch test cherry pick to branch 21 (#32611)
* [test](neredis) Add tpch test for query rewrite by materialized view (#30870)

query rewrite by materialized view sql is as following
q1, q5, q6, q8, q9, q12, q14
the other is not supported now, will be supported later

* change code usage
2024-03-22 15:20:38 +08:00
2e25b0079b fix compile 2024-03-22 09:17:40 +08:00
e5d7ff64aa [fix](backup) clear snapshot info for cancelled backup job to reduce log size (#32604) 2024-03-22 08:53:17 +08:00
6b54171778 [bugfix](deadlock) pipelinex map lock should only scope in map not about pipelinectx's cancel method (#32622)
both global lock in fragment mgr should only protect the map logic, could not use it to protect cancel method.
fragment ctx cancel method should be protected by a lock.
query ctx cancel --> pipelinex fragment cancel ---> query ctx cancel will dead lock.
2024-03-22 08:52:38 +08:00
06026b2f46 [opt](paimon) add split num and partition num (#32597)
1. And selected partition number indicator in PaimonScanNode in explain result.
2. Add raw file split and total split indicator in PaimonScanNode in explain result.
3. Opt the error msg of table valued function, print the error msg first to avoid the long msg be truncated.
2024-03-22 08:52:16 +08:00
8d6561b32d [opt](jdbc catalog) set jdbc test_connection default true (#32546) 2024-03-22 08:52:16 +08:00
60eeff8e18 [enhance](mtmv)refresh mtmv must add auto (#32522) 2024-03-22 08:52:16 +08:00
d7a3ff1ddf [Fix](Outfile) Fix the column type mapping in the orc/parquet file format (#32281)
| Doris Type             | Orc Type                     |  Parquet Type                |
|---------------------|--------------------|------------------------|
| Date                            | Long (logical: DATE)                 |       int32 (Logical: Date)                                        |
| DateTime                    | TIMESTAMP (logical: TIMESTAMP)    |       int96                          |
2024-03-22 08:52:16 +08:00
ca05129fa1 [fix](auth)node priv can login web ui (#32521) 2024-03-22 08:52:16 +08:00
c1d42eaa86 [fix](mtmv)resolve the issue of table version updates in concurrent situations (#32487)
Move the logic for version+1 from `unprotectedCommitTransaction `to`FinishTransaction`, as the write lock for the table was obtained in `FinishTransaction`
2024-03-22 08:52:16 +08:00
fd0bc720e9 [opt](information_schema) Add DEFAULT_ENCRYPTION column to schemata table (#32501) 2024-03-22 08:52:16 +08:00
886aabda31 [Fix](executor) Fix insert select not close coordinator #32571 2024-03-22 08:52:16 +08:00
b727fd6594 (kerberos)fix hive keberos principal usage 2024-03-22 08:16:59 +08:00
0cde0cbf19 (invert index) modify of time series compaction policy 2024-03-22 08:16:30 +08:00
4c8aaa156a [fix](jni) remove 'push_down_predicates' and fix BE crash with decimal predicate (#32253) (#32599) 2024-03-21 14:07:50 +08:00
dea6859e0d [refactor](jdbc catalog) refactor jdbc catalog get databases logic (#32579) 2024-03-21 14:07:50 +08:00
d2968dcf99 [fix](jdbc catalog) Fixed the problem that the metadata_refresh_interval_sec properties cannot be set (#32516) 2024-03-21 14:07:49 +08:00
085696744d [Enhancement] when partition column is datetime, date can work in create table command (#32335) 2024-03-21 14:07:49 +08:00
8e3d28b93a [Feature] Support Array Type compare function for nereids planner (#31701)
Support Array Type compare function for nereids planner
2024-03-21 14:07:49 +08:00
09be4dc7ee [fix](random-bucket) tabletindex when there is no cached value in memory (#32336)
1. In cloud mode, get visible version is a rpc to metaservice, while
loads would get visible version for all partitions.
2. VunionNode should follow batch size.
2024-03-21 14:07:49 +08:00
95322e2ebe [opt](variable) user variable support expression rather than literal (#32492) 2024-03-21 14:07:49 +08:00
2a1d2ad647 [fix](Nereids) decimalv2 and int/date bit arithmetic need cast to bigint (#32311) 2024-03-21 14:07:49 +08:00
b92a764665 [feature](function) Support for aggregate function foreach combiner for some error function (#31913)
Support for aggregate function foreach combiner for some error function
2024-03-21 14:07:49 +08:00
22782fec92 [opt](routine-load) optimize allocate task to be algorithm for load balance (#32021) 2024-03-21 14:07:49 +08:00
6871c964af [fix](nereids)NullSafeEqualToEqual rule only change to equal if both children are not nullable (#32374)
NullSafeEqualToEqual rule only change to equal if both children are not nullable
2024-03-21 14:07:49 +08:00
1b0c4d3aa2 [chor](log) Change log4j rollover strategy to 'max' (#32116)
The log file names are in the format fe.log.${date}-${index}, e.g.
```
fe.log.20240311-1
fe.log.20240311-2
fe.log.20240311-3
...
```
In the previous, fe.log/fe.audit.log will be renamed to fe.log.xxx-1/fe.audit.log.xxx-1
with the minimum index `1` when they reach rotation size. e.g.
```
fe.log.20240311-1 -> fe.log.20240311-2
fe.log.20240311-2 -> fe.log.20240311-3
fe.log.20240311-3 -> fe.log.20240311-4
fe.log            -> fe.log.20240311-1
```

there are some drawbacks
1. log4j needs to rename all existing rotated log to spare the
   index `1`
2. it's hard to deal with the duplicated names, because every time the
   log rotates, it starts from index `1`

After this change, the log will rename to a file name with larger index
instead of `1`. e.g. when rotate
```
fe.log.20240311-1
fe.log.20240311-2
fe.log.20240311-3
fe.log -> fe.log.20240311-4
```
2024-03-21 14:07:49 +08:00