Commit Graph

19455 Commits

Author SHA1 Message Date
4f03bde1e8 [Feat](job)scheduled job allows the time to be set in the past (#36732) (#37573)
## Proposed changes

In some cases, users want to set the start time to a time in the past.

**For periodic scheduling tasks only, the start time of a one-time task
must be set to the future or present time**
**If the start time is the past time, then the next execution time
should be the start time + time interval.**

(cherry picked from commit fa6061070f6d4063b1966e99fc285e4e6b9c3750)



#36732
2024-07-18 19:15:10 +08:00
bd00f56b10 [fix](auth)fix mtmv name to resolve conflicts (#38057)
pick from master #36958 

## Proposed changes

[fix](auth)fix mtmv name to resolve conflicts
2024-07-18 19:13:55 +08:00
ff593806b8 [test](mtmv)add increment create mtmv case (#38054)
pick from master #35707 

## Proposed changes

[test](mtmv)add increment create mtmv case
2024-07-18 19:10:14 +08:00
13e9842f17 [enhancement](memtable) use shared ptr for flush token since it is shared between memtable write thread and flush thread (#38023) (#38068)
pick https://github.com/apache/doris/pull/38023
---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-07-18 19:09:40 +08:00
b7c038eb11 [fix]fix be core when migration tablet to other disk (#37712) (#38062)
![image](https://github.com/user-attachments/assets/01f67160-3ebe-41a1-ac79-a7173d14605c)
The asynchronous task reference captures a local variable
Issue Number: close #36809

Co-authored-by: z404289981 <123865460+z404289981@users.noreply.github.com>
Co-authored-by: huronghui <huronghui@huronghuideMac-mini.local>
2024-07-18 17:51:32 +08:00
10a8cf6fec [test](mtmv)add join infer and derive test case (#38053)
pick from master #32860 

## Proposed changes

[test](mtmv)add join infer and derive test case
2024-07-18 17:50:23 +08:00
8c6ff22e04 [Pick](Variant) fix heap use after free and optimize cases #37991 #37976 (#38037) 2024-07-18 16:53:09 +08:00
88d771d360 [pipeline](fix) Avoid to use a freed dependency when cancelled (#34584) (#38046)
## Proposed changes

pick #34584
<!--Describe your changes.-->
2024-07-18 15:27:10 +08:00
c30c1d2436 [branch-2.1] Picks "[opt](delete) Delete job should retry for failure that is not DELETE_INVALID_XXX #37834" (#38032)
## Proposed changes

picks https://github.com/apache/doris/pull/37834 and
https://github.com/apache/doris/pull/38043
2024-07-18 14:50:30 +08:00
7dd01b1fa9 [fix](publish) Pick Catch exception in genPublishTask to make one failed txn does not block the other txns (#37724) (#38042)
Pick https://github.com/apache/doris/pull/37724
2024-07-18 14:15:49 +08:00
ea457aad77 [fix](Nereids) add order_qt in fuction groovy (#37542) (#38040)
## Proposed changes

#37542
2024-07-18 14:15:07 +08:00
20ee38ea52 [fix](case)fix s3 tvf case failed in pipeline of 2.1 (#37995)
default value `enable_fallback_to_original_planner` is true, resource
priv check only support in nereids,so need set
enable_fallback_to_original_planner=false;
2024-07-18 10:23:40 +08:00
0e248e3594 [fix](inverted index) Corrected the issue of no_index_match failure caused by empty data #37947 (#38002) 2024-07-18 10:04:36 +08:00
ebc178af23 [fix](nereids)acos function should return null literal instead of NaN value (#37996)
pick from master https://github.com/apache/doris/pull/37932

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-18 09:28:56 +08:00
dcfc72bb36 [fix](nereids) bug: after is-null stats derive, other column stats are dropped (#37809) (#38024)
## Proposed changes
pick from #37809 
Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-18 08:11:09 +08:00
b16cd30a20 [opt](inverted index) set support_phrase default true if parser is set (#38028)
## Proposed changes

https://github.com/apache/doris/pull/37949

<!--Describe your changes.-->
2024-07-18 08:07:16 +08:00
38885d4b00 [fix](load) fix memtable agg functions (#38017) (#38021)
backport #38017
2024-07-17 23:04:57 +08:00
2dbc996a1b [fix](regression)Fix test_agg_complex_type fail case. Improve analyze_stats time consumption. (#37895) (#38014)
backport: https://github.com/apache/doris/pull/37895
2024-07-17 22:15:21 +08:00
3d5043817a Revert "[opt](serde)Optimize the filling of fixed values ​​into block columns without repeated deserialization. (#37377)" (#38007)
Reverts apache/doris#37530
Need more test, revert it temporarily
2024-07-17 21:44:25 +08:00
629d7ff048 [fix](nereids) fix aggr node colocate flag local shuffle depends on (#38016)
## Proposed changes

pick from https://github.com/apache/doris/pull/37788

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
2024-07-17 21:34:30 +08:00
78d51ca320 [fix](Nereids) tablet prune wrong when decimal value scale is nagtive (#37889) (#38013)
pick from master #37889

we use unscaled value of BigDecimal in tablet prune. So we need to
ensure BigDecimal's precision and scale is same with the literal
contains it.
2024-07-17 20:12:14 +08:00
7180290d68 [fix](dynamic partition) fix create dynamic partition with overlap not throw exception #37924 (#37964)
cherry-pick:  #37924
2024-07-17 19:13:35 +08:00
4d359cd946 [improvement](regression)Use createMV to create materialized view. (#37837) (#37992)
backport: https://github.com/apache/doris/pull/37837
2024-07-17 17:16:04 +08:00
2e6f49af15 [opt](Nereids) support no-key hint parameter (#37720) (#37988)
pick from master #37720

support hint use parameter without key, like:

```sql
SELECT /*+ query_timeout(3000) */ * FROM t;
```
2024-07-17 16:59:25 +08:00
b2a4cff51b [fix](nereids)fix nullable property of ForEachCombinator (#37980)
## Proposed changes

pick from master https://github.com/apache/doris/pull/37796

<!--Describe your changes.-->
2024-07-17 13:48:21 +08:00
1875267796 [fix](routine-load) fix routine load pause when Kafka data deleted after TTL (#37288) (#37983)
pick (#37288)

When using routine load, After the data load is completed, the lag is
still a positive number:
```
  Lag: {"0":16,"1":15,"2":16,"3":16,"4":16,"5":16,"6":15,"7":16,"8":16,"9":16,"10":15,"11":16,"12":15,"13":15,"14":16,"15":16,"16":17,"17":15,"18":16,"19":15,"20":16,"21":16,"22":16,"23":16,"24":15,"25":17,"26":17,"27":16,"28":16,"29":16,"30":16,"31":17,"32":14,"33":16,"34":17,"35":16,"36":15,"37":15,"38":15,"39":16,"40":16,"41":16,"42":15,"43":15,"44":17,"45":16,"46":15,"47":15,"48":16,"49":17,"50":16,"51":15,"52":16,"53":15,"54":15,"55":17,"56":16,"57":17,"58":16,"59":16,"60":15,"61":15,"62":16,"63":16,"64":17,"65":16,"66":15,"67":16,"68":17,"69":16,"70":15,"71":17}
```
and the routing load is paused when the Kafka data reaches TTL and is
deleted, the error is `out of range`.

The reason why this happened is EOF has it offset which needed
statistics.

**note(important):**
After the bug is fixed, if you set 
```
"property.enable.partition.eof" = "false"
```
in your routine load job, it will meet the problem. For EOF has offset,
and the config is true in Doris default.
2024-07-17 13:47:26 +08:00
Pxl
db0a43bad2 [Chore](exchange) change LocalExchangeSharedState:mem_usage signed ty… (#37981)
pick from #36682
2024-07-17 13:46:51 +08:00
f2cf8b7d6f [Fix](InternalSchema) Compute nodes should not be used for Internal schema three replica (#36130) (#37961)
bp #36130

Co-authored-by: HB <137497191@qq.com>
Co-authored-by: camby <104178625@qq.com>
2024-07-17 13:45:32 +08:00
cf5c555223 [fix](case)Fix unusual case in manager_interface.(#37930) (#37967)
bp #37930 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-17 12:27:07 +08:00
33b379a51d [bug](join) remove broadcast join check about shared hashtable signal (#37969)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-17 12:26:19 +08:00
ba66ff5768 [fix](multi-catalog)fix paimon meta properties convert (#37249) (#37958)
bp #37249

Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
2024-07-17 01:04:33 +08:00
b6e5281a1c [Fix](bug) fix the divide zero in local shuffle: (#37948)
## Proposed changes

cherry pick #37906 

<!--Describe your changes.-->
2024-07-17 01:03:53 +08:00
21c6b854f7 [fix](explode-json-object)fix explode json object (#37956)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-17 01:03:07 +08:00
3d095c737d [fix](case)partition num will be 3 in daily pipeline (#37954)
pick:https://github.com/apache/doris/pull/37945
2024-07-17 00:59:15 +08:00
dbd2b0abf7 [improvement](mtmv) improve mv rewrite performance by reuse the shuttled expression (#37197) (#37935)
## Proposed changes

chrry-pick 2.1
pr: https://github.com/apache/doris/pull/37197
commitId: 701c7db4
2024-07-17 00:52:58 +08:00
359e50fc58 [fix](load) change tablet schema pointer to shared_ptr in memtable (#37927) (#37939)
backport #37927
2024-07-16 22:32:03 +08:00
f7068b5658 [cherry-pick](branch-2.1) Make doris read hive text table parameters and behavior consistent with hive (#37840)
## Proposed changes

pick from master https://github.com/apache/doris/pull/37638

<!--Describe your changes.-->
2024-07-16 22:24:50 +08:00
49faccbf50 [enhancement](nereids) speedup sql cache with use variable as partition predicate (#37943)
follow up #37090

support reuse sql cache when use variable as partition predicate and variable change:
```sql
set @dt='2024-07-16';
-- create cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-17';
-- create cache 2, will not invalidate cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-16';
-- reuse cache 1
select * from tbl where dt = @dt;
```
2024-07-16 22:11:18 +08:00
b15ccdbe98 [Pick](Variant) pick some fix (#37922)
#37674
#37839
#37883 
#37857 
#37794
2024-07-16 21:38:47 +08:00
2edb9501b5 [Pick]Fix show role stmt missing grouo info (#37920)
## Proposed changes

pick #36032
2024-07-16 20:41:59 +08:00
cc85f7b94c [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) (#37897) 2024-07-16 20:32:30 +08:00
adf11737f0 [fix](catalog)Fix internal program error causing client to get stuck … (#37821)
…(#37551)

pick: https://github.com/apache/doris/pull/37551
2024-07-16 20:22:47 +08:00
9ff129b630 [fix](stream_load) fix stream load may failed caused by column name with keyword (#35822) (#37890)
#35938 #35822 
let 
KW_SQL,
KW_CACHE,
KW_COLOCATE,
KW_COMPRESS_TYPE,
KW_DORIS_INTERNAL_TABLE_ID,
KW_HOTSPOT,
KW_PRIVILEGES,
KW_RECENT,
KW_STAGES,
KW_WARM,
KW_UP,
KW_CONVERT_LSC
be as non-reserved

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

---------

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2024-07-16 20:20:34 +08:00
36337c8bd9 [fix](multicast) should not ignore Status of block::merge #35886 (#37869)
## Proposed changes

BP #35886
2024-07-16 19:03:24 +08:00
faa425bee5 [fix](colocate) fix colocate join while multi tables (#37729) (#37859)
cherry-pick #37729 to branch-2.1
2024-07-16 19:02:36 +08:00
8440303b91 [fix](delete) Incorrect precision detection for the decimal type in condition.​ (#37801) (#37904)
## Proposed changes

pick #37801

For precision like Decimal(7,7), the value "0.1234567" should be
valid(the integer part is 0).

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 19:02:02 +08:00
9861f81630 [branch-2.1](memory) Fix Jemalloc Cache Memory Tracker (#37905)
pick #37464
2024-07-16 19:01:31 +08:00
81a7542cae [pick]Add audit log event queue size limit (#37914)
## Proposed changes
pick #37786
2024-07-16 19:00:22 +08:00
cc6ff12097 [opt](function) Optimize the trim function for single-char inputs (#3… (#37799)
https://github.com/apache/doris/pull/36497

before
```
mysql [test]>select count(ltrim(str,"1")) from stringDb2;
+------------------------+
| count(ltrim(str, '1')) |
+------------------------+
|               64000000 |
+------------------------+
1 row in set (7.79 sec)
```

now
```
mysql [test]>select count(ltrim(str,"1")) from stringDb2;
+------------------------+
| count(ltrim(str, '1')) |
+------------------------+
|               64000000 |
+------------------------+
1 row in set (0.73 sec)
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 17:52:52 +08:00
638ee2607e [fix](regression test) fix test_schema_change_fail due to set force_olap_table_replication_num #34343 (#37860)
cherry pick from #34343
2024-07-16 17:40:42 +08:00