Commit Graph

7259 Commits

Author SHA1 Message Date
2edb9501b5 [Pick]Fix show role stmt missing grouo info (#37920)
## Proposed changes

pick #36032
2024-07-16 20:41:59 +08:00
adf11737f0 [fix](catalog)Fix internal program error causing client to get stuck … (#37821)
…(#37551)

pick: https://github.com/apache/doris/pull/37551
2024-07-16 20:22:47 +08:00
faa425bee5 [fix](colocate) fix colocate join while multi tables (#37729) (#37859)
cherry-pick #37729 to branch-2.1
2024-07-16 19:02:36 +08:00
81a7542cae [pick]Add audit log event queue size limit (#37914)
## Proposed changes
pick #37786
2024-07-16 19:00:22 +08:00
638d4b6f27 [Bug](function) Fix function for cast string as date/datetime (#35637) (#37891)
## Proposed changes

Issue Number: close #35635
cherry-pick https://github.com/apache/doris/pull/35637 from master to
branch-2.1

<!--Describe your changes.-->
Cast rules:Consistent with mysql.
String:Date
The first part is 1-digit x: 000x-month-day
The first part is 2-digit xy: 20xy-month-day / 19xy-month-day The first
part is 3-digit xyz: 20xy-0z-day / 19xy-0z-day The first part is 4-digit
xyzw: xyzw-month-day

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 17:22:03 +08:00
9fd4795080 [opt](iceberg)Add a new appearance to display the pushDown count for 2.1 (#37046) (#34928) (#37810)
## Proposed changes

bp: #37046 #34928
2024-07-16 16:03:44 +08:00
cb2f95ee2e [fix](Nereids) fix fe fold constant failed when using like function (#37864)
cherry-pick: #37616
2024-07-16 16:02:29 +08:00
80ea98b371 [fix](nereids)subquery unnesting get wrong result if correlated conjuncts is not slot_a = slot_b (#37683)
pick from master https://github.com/apache/doris/pull/37644

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 15:06:40 +08:00
2c80259f66 [fix](mtmv) use isManagedTable instead of check table type (#34287) (#37822)
pick: https://github.com/apache/doris/pull/34287
2024-07-16 15:01:28 +08:00
2c4b5519e9 [cherry-pick](branch-2.1) let insert statement support CTE (#36150) (#36265)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

cherry-pick: #36150
2024-07-16 14:50:53 +08:00
e84b9a0eaa [fix](auth)fix fe can not restart when replay create row policy log (… (#37820)
pick: https://github.com/apache/doris/pull/37342
2024-07-16 11:28:19 +08:00
2957fdc039 [branch2.1] pick [fix](show) show create table show index comment err… (#37034)
## Proposed changes
pick https://github.com/apache/doris/pull/36306
2024-07-16 11:19:27 +08:00
f1d22a9610 [fix](mtmv) fix mtmv task nereids cost too much time (#37589) (#37819)
pick: https://github.com/apache/doris/pull/37589
2024-07-16 11:08:18 +08:00
3cb1d4e842 [feature](json)support explode_json_object func #36887 (#37378) 2024-07-16 10:59:11 +08:00
e7a001c420 [enhance](mtmv)support partition tvf (#37795)
pick from: https://github.com/apache/doris/pull/36479 and
https://github.com/apache/doris/pull/37201
2024-07-16 09:27:44 +08:00
63c2d22513 [cherry-pick](branch-2.1) Pick "[Fix](delete command) Mark delete sign when do delete command in MoW table (#35917)" (#37594)
Pick #35917 and #37151
2024-07-15 18:54:01 +08:00
03e21dddff [cherry-pick](branch-21) fix cast string to int return wrong result (#36788) (#37803)
## Proposed changes
cherry-pick from master:
https://github.com/apache/doris/pull/36788
https://github.com/apache/doris/pull/36505

<!--Describe your changes.-->
2024-07-15 18:48:49 +08:00
Pxl
d1fc4e2e60 [Bug](query) fix meet invalid column when direct scan on mow mv (#37806)
pick from #36483
2024-07-15 18:29:30 +08:00
57301920e3 [fix](colocate join) fix wrong use of colocate join (#37361) (#37714)
cherry-pick from master #37361
2024-07-15 16:47:17 +08:00
e5339a4014 [feature](ES Catalog)Support control scroll level by config #37180 (#37290)
## Proposed changes

backport #37180
2024-07-15 16:41:38 +08:00
ff7a04093e [fix](fe) fix several blocking bugs #37756 (#37757)
bp #37756
2024-07-15 15:56:01 +08:00
2759383365 [branch-2.1](timezone) refactor tzdata load to accelerate and unify timezone parsing (#37062) (#37269)
pick https://github.com/apache/doris/pull/37062

1. revert https://github.com/apache/doris/pull/25097. we decide to rely
on OS. not maintain independent tzdata anymore to keep result
consistency
2. refactor timezone load. removed rwlock.

before:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (6.88 sec)
```
now:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (2.61 sec)
```
3. now don't support timezone offset format string like 'UTC+8', like we
already said in
https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage
4. support case-insensitive timezone parsing in nereids.
5. a bug when parse timezone using nereids. should check DST by input,
but wrongly by now before. now fixed.

doc pr: https://github.com/apache/doris-website/pull/810
2024-07-15 10:56:48 +08:00
3da5b17abf [branch-2.1](timezone) make TimeUtils formatter use correct time_zone (#37465) (#37652)
All timestamp/datetime parsing in Doris is controlled by the session
variable `time_zone`.
Apply it also to interface of `TimeUtils` in FE.

pick https://github.com/apache/doris/pull/37465
2024-07-15 10:23:38 +08:00
16de141743 [regression](kerberos)add hive kerberos docker regression env (#37657)
## Proposed changes
pick:
[regression](kerberos)fix regression pipeline env when write hosts 
(#37057)
[regression](kerberos)add hive kerberos docker regression env (#36430)
2024-07-15 09:35:39 +08:00
ec8467f57b [fix](auto bucket) Fix hit not support alter estimate_partition_size #33670 (#37633)
cherry pick from #33670
2024-07-13 22:12:38 +08:00
8930df3b31 [Feature](iceberg-writer) Implements iceberg partition transform. (#37692)
## Proposed changes

Cherry-pick iceberg partition transform functionality. #36289 #36889

---------

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
Co-authored-by: lik40 <lik40@chinatelecom.cn>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
2024-07-13 16:07:50 +08:00
56a207c3f0 [case](paimon/iceberg)move cases from p2 to p0 (#37276) (#37738)
bp #37276

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-07-13 10:01:05 +08:00
20758576b2 [fix](split) remove retry when fetch split batch failed (#37637)
bp: #37636
2024-07-12 22:46:03 +08:00
019cd9b4ec [fix](hudi) return empty if there is no commit implemented (#37703)
bp: #37702
2024-07-12 22:44:58 +08:00
f2556ba182 [feature](insert)support external hive truncate table DDL (#37659)
pick: #36801
2024-07-12 22:37:47 +08:00
259d28407e [improvement](statistics)Enable estimate hive table row count using file size. (#37218) (#37694)
backport: https://github.com/apache/doris/pull/37218
2024-07-12 13:47:27 +08:00
ffa9e49bc7 [feature](mtmv) pick some mtmv pr from master (#37651)
cherry-pick from master
pr: #36318
commitId: c1999479

pr: #36111
commitId: 35ebef62

pr: #36175
commitId: 4c8e66b4

pr: #36414
commitId: 5e009b5a

pr: #36770
commitId: 19e2126c

pr: #36567
commitId: 3da83514
2024-07-12 10:35:54 +08:00
217eac790b [pick](Variant) pick some refactor and fix #34925 #36317 #36201 #36793 (#37526) 2024-07-11 21:25:34 +08:00
fdf21ec251 [fix](readconsistency) avoid table not exist error (#37593) (#37641)
Query following createting table would throw table not exist error.

For example.
t1: client issue create table to master fe
t2: client issue query sql to observer fe, the query would fail due to
not exist table in plan phase.
t3: observer fe receive editlog creating the table from the master fe

After the pr:
query at t2 would wait until latest edit log is received from master fe
in the observer fe.

pick #37593

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-11 18:57:53 +08:00
cee3cf8499 [fix](statistics)Fix column cached stats size bug. (#37545) (#37667)
backport: https://github.com/apache/doris/pull/37545
2024-07-11 18:53:12 +08:00
8a0d940914 [fix](publish) Pick Fix publish failed because because task is null (#37546)
## Proposed changes

Pick https://github.com/apache/doris/pull/37531

This pr catch the exception to make the failed txn does not block the
other txns.
2024-07-11 15:22:04 +08:00
1eb04cf538 [feature](mtmv) Support query rewrite by materialized view when query is aggregate and materialized view has no aggregate (#36278) (#37497)
cherry-pick from master
pr: #36278
commitId: 649f9bc6
2024-07-11 10:54:50 +08:00
e6b8ebc847 [Fix](Short Circuit) fix no project list in OlapScanNode (#37121) (#37504)
pick from #37121
2024-07-11 10:04:28 +08:00
e1cb568d11 [Optimize] Add session variable `max_fetch_remote_schema_tablet_count… (#37505)
pick from #37217
2024-07-11 10:04:20 +08:00
f58032f1da [fix](dynamic partition) drop partition exclude history_partition_num #37539 (#37570)
cherry pick from #37539

---------

Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2024-07-10 22:09:00 +08:00
a7416f6ff9 [fix](dump) do not report success if dumping fails (#37510)
## Proposed changes
pick https://github.com/apache/doris/pull/37508

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-10 16:26:14 +08:00
db4d061a68 [fix](Nereids) null type result with alias name should keep alias name (#37457) (#37524)
pick from master #37457
2024-07-09 20:46:51 +08:00
8ef83259ff [fix](planner) fix bug of select stmt toSql(#37274) (#37344)
cherry-pick from master #37274
2024-07-09 20:34:57 +08:00
9b075bc873 [fix](nereids) derive column stats for 'expr and A is not null' (#37235) (#37498)
pick from #37235 
the algorithm for computing stats for "expr1 and expr2" predicate is as
following:
1. compute output stats of expr1 based on input stats. the result stats
is denoted by leftStats
2. compute stats of expr2 based on leftStats after step1, leftStats
should be normalized to avoid abnormal cases, such as ndv > rowCount or
numNulls > rowCount

Issue Number: close #xxx

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-09 17:46:57 +08:00
7f6b846f58 [improve](fe) Add a counter metric for recording large editlog write (#37328) (#37474) 2024-07-09 17:16:31 +08:00
9b500faa0c [fix](create table) create table fail not write drop table editlog #37488 (#37506)
cherry pick from #37488
2024-07-09 13:44:43 +08:00
4426d6d80f [fix](fe) Add check editlog size mechanism for backupJob (#35653) (#37466)
* When creating a backupJob with huge of tables in a database, it can
cause backupJob editlog size over 2GB and bdbje will throw exception
because of ByteBuffer overflow

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-09 10:33:28 +08:00
ca0e44f83f [fix](case) fix struct format out files (#37350) (#37499)
bp #37350
2024-07-09 10:11:50 +08:00
a6ec78ec5f [fix](truncate) fix tablet invert index leaky #37334 (#37410)
cherry pick from #37334
2024-07-08 20:57:30 +08:00
4c5a7b26e2 [chore](restore) Log partition visible version (#37414)
Cherry-pick #36920.
2024-07-08 18:58:16 +08:00