Commit Graph

7271 Commits

Author SHA1 Message Date
dcfc72bb36 [fix](nereids) bug: after is-null stats derive, other column stats are dropped (#37809) (#38024)
## Proposed changes
pick from #37809 
Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-18 08:11:09 +08:00
b16cd30a20 [opt](inverted index) set support_phrase default true if parser is set (#38028)
## Proposed changes

https://github.com/apache/doris/pull/37949

<!--Describe your changes.-->
2024-07-18 08:07:16 +08:00
629d7ff048 [fix](nereids) fix aggr node colocate flag local shuffle depends on (#38016)
## Proposed changes

pick from https://github.com/apache/doris/pull/37788

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
2024-07-17 21:34:30 +08:00
78d51ca320 [fix](Nereids) tablet prune wrong when decimal value scale is nagtive (#37889) (#38013)
pick from master #37889

we use unscaled value of BigDecimal in tablet prune. So we need to
ensure BigDecimal's precision and scale is same with the literal
contains it.
2024-07-17 20:12:14 +08:00
7180290d68 [fix](dynamic partition) fix create dynamic partition with overlap not throw exception #37924 (#37964)
cherry-pick:  #37924
2024-07-17 19:13:35 +08:00
2e6f49af15 [opt](Nereids) support no-key hint parameter (#37720) (#37988)
pick from master #37720

support hint use parameter without key, like:

```sql
SELECT /*+ query_timeout(3000) */ * FROM t;
```
2024-07-17 16:59:25 +08:00
b2a4cff51b [fix](nereids)fix nullable property of ForEachCombinator (#37980)
## Proposed changes

pick from master https://github.com/apache/doris/pull/37796

<!--Describe your changes.-->
2024-07-17 13:48:21 +08:00
f2cf8b7d6f [Fix](InternalSchema) Compute nodes should not be used for Internal schema three replica (#36130) (#37961)
bp #36130

Co-authored-by: HB <137497191@qq.com>
Co-authored-by: camby <104178625@qq.com>
2024-07-17 13:45:32 +08:00
ba66ff5768 [fix](multi-catalog)fix paimon meta properties convert (#37249) (#37958)
bp #37249

Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
2024-07-17 01:04:33 +08:00
dbd2b0abf7 [improvement](mtmv) improve mv rewrite performance by reuse the shuttled expression (#37197) (#37935)
## Proposed changes

chrry-pick 2.1
pr: https://github.com/apache/doris/pull/37197
commitId: 701c7db4
2024-07-17 00:52:58 +08:00
f7068b5658 [cherry-pick](branch-2.1) Make doris read hive text table parameters and behavior consistent with hive (#37840)
## Proposed changes

pick from master https://github.com/apache/doris/pull/37638

<!--Describe your changes.-->
2024-07-16 22:24:50 +08:00
49faccbf50 [enhancement](nereids) speedup sql cache with use variable as partition predicate (#37943)
follow up #37090

support reuse sql cache when use variable as partition predicate and variable change:
```sql
set @dt='2024-07-16';
-- create cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-17';
-- create cache 2, will not invalidate cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-16';
-- reuse cache 1
select * from tbl where dt = @dt;
```
2024-07-16 22:11:18 +08:00
2edb9501b5 [Pick]Fix show role stmt missing grouo info (#37920)
## Proposed changes

pick #36032
2024-07-16 20:41:59 +08:00
adf11737f0 [fix](catalog)Fix internal program error causing client to get stuck … (#37821)
…(#37551)

pick: https://github.com/apache/doris/pull/37551
2024-07-16 20:22:47 +08:00
faa425bee5 [fix](colocate) fix colocate join while multi tables (#37729) (#37859)
cherry-pick #37729 to branch-2.1
2024-07-16 19:02:36 +08:00
81a7542cae [pick]Add audit log event queue size limit (#37914)
## Proposed changes
pick #37786
2024-07-16 19:00:22 +08:00
638d4b6f27 [Bug](function) Fix function for cast string as date/datetime (#35637) (#37891)
## Proposed changes

Issue Number: close #35635
cherry-pick https://github.com/apache/doris/pull/35637 from master to
branch-2.1

<!--Describe your changes.-->
Cast rules:Consistent with mysql.
String:Date
The first part is 1-digit x: 000x-month-day
The first part is 2-digit xy: 20xy-month-day / 19xy-month-day The first
part is 3-digit xyz: 20xy-0z-day / 19xy-0z-day The first part is 4-digit
xyzw: xyzw-month-day

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 17:22:03 +08:00
9fd4795080 [opt](iceberg)Add a new appearance to display the pushDown count for 2.1 (#37046) (#34928) (#37810)
## Proposed changes

bp: #37046 #34928
2024-07-16 16:03:44 +08:00
cb2f95ee2e [fix](Nereids) fix fe fold constant failed when using like function (#37864)
cherry-pick: #37616
2024-07-16 16:02:29 +08:00
80ea98b371 [fix](nereids)subquery unnesting get wrong result if correlated conjuncts is not slot_a = slot_b (#37683)
pick from master https://github.com/apache/doris/pull/37644

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 15:06:40 +08:00
2c80259f66 [fix](mtmv) use isManagedTable instead of check table type (#34287) (#37822)
pick: https://github.com/apache/doris/pull/34287
2024-07-16 15:01:28 +08:00
2c4b5519e9 [cherry-pick](branch-2.1) let insert statement support CTE (#36150) (#36265)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

cherry-pick: #36150
2024-07-16 14:50:53 +08:00
e84b9a0eaa [fix](auth)fix fe can not restart when replay create row policy log (… (#37820)
pick: https://github.com/apache/doris/pull/37342
2024-07-16 11:28:19 +08:00
2957fdc039 [branch2.1] pick [fix](show) show create table show index comment err… (#37034)
## Proposed changes
pick https://github.com/apache/doris/pull/36306
2024-07-16 11:19:27 +08:00
f1d22a9610 [fix](mtmv) fix mtmv task nereids cost too much time (#37589) (#37819)
pick: https://github.com/apache/doris/pull/37589
2024-07-16 11:08:18 +08:00
3cb1d4e842 [feature](json)support explode_json_object func #36887 (#37378) 2024-07-16 10:59:11 +08:00
e7a001c420 [enhance](mtmv)support partition tvf (#37795)
pick from: https://github.com/apache/doris/pull/36479 and
https://github.com/apache/doris/pull/37201
2024-07-16 09:27:44 +08:00
63c2d22513 [cherry-pick](branch-2.1) Pick "[Fix](delete command) Mark delete sign when do delete command in MoW table (#35917)" (#37594)
Pick #35917 and #37151
2024-07-15 18:54:01 +08:00
03e21dddff [cherry-pick](branch-21) fix cast string to int return wrong result (#36788) (#37803)
## Proposed changes
cherry-pick from master:
https://github.com/apache/doris/pull/36788
https://github.com/apache/doris/pull/36505

<!--Describe your changes.-->
2024-07-15 18:48:49 +08:00
Pxl
d1fc4e2e60 [Bug](query) fix meet invalid column when direct scan on mow mv (#37806)
pick from #36483
2024-07-15 18:29:30 +08:00
57301920e3 [fix](colocate join) fix wrong use of colocate join (#37361) (#37714)
cherry-pick from master #37361
2024-07-15 16:47:17 +08:00
e5339a4014 [feature](ES Catalog)Support control scroll level by config #37180 (#37290)
## Proposed changes

backport #37180
2024-07-15 16:41:38 +08:00
ff7a04093e [fix](fe) fix several blocking bugs #37756 (#37757)
bp #37756
2024-07-15 15:56:01 +08:00
2759383365 [branch-2.1](timezone) refactor tzdata load to accelerate and unify timezone parsing (#37062) (#37269)
pick https://github.com/apache/doris/pull/37062

1. revert https://github.com/apache/doris/pull/25097. we decide to rely
on OS. not maintain independent tzdata anymore to keep result
consistency
2. refactor timezone load. removed rwlock.

before:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (6.88 sec)
```
now:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (2.61 sec)
```
3. now don't support timezone offset format string like 'UTC+8', like we
already said in
https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage
4. support case-insensitive timezone parsing in nereids.
5. a bug when parse timezone using nereids. should check DST by input,
but wrongly by now before. now fixed.

doc pr: https://github.com/apache/doris-website/pull/810
2024-07-15 10:56:48 +08:00
3da5b17abf [branch-2.1](timezone) make TimeUtils formatter use correct time_zone (#37465) (#37652)
All timestamp/datetime parsing in Doris is controlled by the session
variable `time_zone`.
Apply it also to interface of `TimeUtils` in FE.

pick https://github.com/apache/doris/pull/37465
2024-07-15 10:23:38 +08:00
16de141743 [regression](kerberos)add hive kerberos docker regression env (#37657)
## Proposed changes
pick:
[regression](kerberos)fix regression pipeline env when write hosts 
(#37057)
[regression](kerberos)add hive kerberos docker regression env (#36430)
2024-07-15 09:35:39 +08:00
ec8467f57b [fix](auto bucket) Fix hit not support alter estimate_partition_size #33670 (#37633)
cherry pick from #33670
2024-07-13 22:12:38 +08:00
8930df3b31 [Feature](iceberg-writer) Implements iceberg partition transform. (#37692)
## Proposed changes

Cherry-pick iceberg partition transform functionality. #36289 #36889

---------

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
Co-authored-by: lik40 <lik40@chinatelecom.cn>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
2024-07-13 16:07:50 +08:00
56a207c3f0 [case](paimon/iceberg)move cases from p2 to p0 (#37276) (#37738)
bp #37276

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-07-13 10:01:05 +08:00
20758576b2 [fix](split) remove retry when fetch split batch failed (#37637)
bp: #37636
2024-07-12 22:46:03 +08:00
019cd9b4ec [fix](hudi) return empty if there is no commit implemented (#37703)
bp: #37702
2024-07-12 22:44:58 +08:00
f2556ba182 [feature](insert)support external hive truncate table DDL (#37659)
pick: #36801
2024-07-12 22:37:47 +08:00
259d28407e [improvement](statistics)Enable estimate hive table row count using file size. (#37218) (#37694)
backport: https://github.com/apache/doris/pull/37218
2024-07-12 13:47:27 +08:00
ffa9e49bc7 [feature](mtmv) pick some mtmv pr from master (#37651)
cherry-pick from master
pr: #36318
commitId: c1999479

pr: #36111
commitId: 35ebef62

pr: #36175
commitId: 4c8e66b4

pr: #36414
commitId: 5e009b5a

pr: #36770
commitId: 19e2126c

pr: #36567
commitId: 3da83514
2024-07-12 10:35:54 +08:00
217eac790b [pick](Variant) pick some refactor and fix #34925 #36317 #36201 #36793 (#37526) 2024-07-11 21:25:34 +08:00
fdf21ec251 [fix](readconsistency) avoid table not exist error (#37593) (#37641)
Query following createting table would throw table not exist error.

For example.
t1: client issue create table to master fe
t2: client issue query sql to observer fe, the query would fail due to
not exist table in plan phase.
t3: observer fe receive editlog creating the table from the master fe

After the pr:
query at t2 would wait until latest edit log is received from master fe
in the observer fe.

pick #37593

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-11 18:57:53 +08:00
cee3cf8499 [fix](statistics)Fix column cached stats size bug. (#37545) (#37667)
backport: https://github.com/apache/doris/pull/37545
2024-07-11 18:53:12 +08:00
8a0d940914 [fix](publish) Pick Fix publish failed because because task is null (#37546)
## Proposed changes

Pick https://github.com/apache/doris/pull/37531

This pr catch the exception to make the failed txn does not block the
other txns.
2024-07-11 15:22:04 +08:00
1eb04cf538 [feature](mtmv) Support query rewrite by materialized view when query is aggregate and materialized view has no aggregate (#36278) (#37497)
cherry-pick from master
pr: #36278
commitId: 649f9bc6
2024-07-11 10:54:50 +08:00
e6b8ebc847 [Fix](Short Circuit) fix no project list in OlapScanNode (#37121) (#37504)
pick from #37121
2024-07-11 10:04:28 +08:00