Commit Graph

8027 Commits

Author SHA1 Message Date
54787f51c7 [log](load) print reason when LoadPathMgr failed to delete file (#38235) (#38340)
backport #38235
2024-07-25 17:28:09 +08:00
4288f09e8c [fix](debug_point) Add debug point:reached_limit_early & return_empty_block (#38328)
## Proposed changes

pick #38127 and #37629
2024-07-25 14:46:40 +08:00
a8139d9f39 [Improvement](profile) Provide more info for schedule time (#38290) (#38338)
## Proposed changes

pick #38290

<!--Describe your changes.-->
2024-07-25 14:43:52 +08:00
e2bb86e7f8 [fix](inverted index) fixed in_list condition not indexed on pipelinex (#38178)
## Proposed changes

https://github.com/apache/doris/pull/36565
https://github.com/apache/doris/pull/37842
https://github.com/apache/doris/pull/37921
https://github.com/apache/doris/pull/37386

<!--Describe your changes.-->
2024-07-25 14:42:34 +08:00
a751372e76 [Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 (#37257)
## Proposed changes

backport #37234
2024-07-25 13:51:59 +08:00
73fc55b203 [Pick](Variant) fix some issue by RQG (#38336)
#38318 
#38291
2024-07-25 12:19:07 +08:00
70cde39fe0 [cherry-pick](branch-21) fix conv function get wrong result as parse overflow (#38001) (#38309)
## Proposed changes

cherry-pick from https://github.com/apache/doris/pull/38001

<!--Describe your changes.-->
2024-07-25 12:06:46 +08:00
e9052e2180 [cherry-pick](branch-21) fix mod function cause core dump (#37999) (#38308)
## Proposed changes
cherry-pick from master https://github.com/apache/doris/pull/37999

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-25 12:06:21 +08:00
57864e8554 [cherry-pick](branch-21) fix collect_set function core dump without arena pool (#38234) (#38307)
## Proposed changes

cherry-pick from master #38234

<!--Describe your changes.-->
2024-07-25 12:05:52 +08:00
d19b3a5cb6 [fix](function) the bucket number arg of width_bucket should be a positive integer value (#37892) (#38295)
## Proposed changes

pick #37892
2024-07-25 11:56:47 +08:00
21b3fc3d1e [branch-2.1](function) fix coredump for MULTI_MATCH_ANY (#37959) (#38314)
pick https://github.com/apache/doris/pull/37959

[INVALID_ARGUMENT][E33] Compile regexp expression failed. got Embedded
start anchors not supported.. some expressions may be illegal
2024-07-25 11:34:22 +08:00
79a6496bb6 [branch-2.1](function) fix wrong result when convert_tz is out of bound (#37358) (#38313)
## Proposed changes

pick https://github.com/apache/doris/pull/37358

before:
```sql
mysql> select CONVERT_TZ(cast('0000-01-01 00:00:00.00001'  as DATETIMEV1), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533)));
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| convert_tz(cast('0000-01-01 00:00:00.00001' as DATETIME), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533))) |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| q535-12-31 08:01:19                                                                                                                               |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.12 sec)
```
now:
```sql
mysql> select CONVERT_TZ(cast('0000-01-01 00:00:00.00001'  as DATETIMEV1), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533)));
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| convert_tz(cast('0000-01-01 00:00:00.00001' as DATETIME), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533))) |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| NULL                                                                                                                                              |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.09 sec)
```
2024-07-25 11:32:44 +08:00
3ea26a8c95 [fix](external) record not found file number (#38253) (#38285)
bp #38253
2024-07-25 11:03:19 +08:00
9a40cd5f9f [fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction (#38224) (#38299)
pick master #38224
2024-07-24 19:03:40 +08:00
ef00dad680 [Fix](multi-catalog) Fix some undefined behaviors. (#38274)
## Proposed changes

backport #37845
2024-07-24 16:14:34 +08:00
c6cd1e2e3d [opt](inverted index) opt value extraction from column to string (#38179)
## Proposed changes

https://github.com/apache/doris/pull/37395
2024-07-24 15:38:41 +08:00
10c5c336d8 [branch-2.1](arrow-flight-sql) Add config arrow_flight_result_sink_buffer_size_rows (#38223)
pick #38221
2024-07-24 15:15:39 +08:00
bb2bc77717 [Pick 2.1](inverted index) skip index compaction for inverted index V2 (#38278)
Enabling index compaction in the inverted index V2 format currently
causes unexpected errors, especially in the case of tables with hybrid
indexes, such as BKD index and Fulltext index together.

backport #38209
2024-07-24 14:38:25 +08:00
792bd7c74a [cherry-pick](branch-2.1) Pick "[Enhancement](group commit)Optimize be select for group commit #35558" (#37830)
Pick #35558
2024-07-24 09:21:07 +08:00
Pxl
cf2120a44a [Chore](brpc) add gc for abafreelist to avoid eagain and set brpc tim… (#38177)
pick from #37888
2024-07-23 21:24:36 +08:00
193be20c86 [feature](csv)Supports reading CSV data using LF and CRLF as line separators. (#37687) (#38099)
bp #37687
2024-07-22 22:53:04 +08:00
aff3f292a7 [fix](pipeline) Fix blocked tasks if query is canceled before opening (#38200)
## Proposed changes

pick #38206

<!--Describe your changes.-->
2024-07-22 18:22:22 +08:00
73199122c2 [enhancement](compaction) Control the parallelism for urgent compacton tasks (#37782) (#38189)
## Proposed changes

For some urgent compaction tasks, their submittion should take
parallelism into account.

Currently, we apply the control policy for data loading in specific.
Other source of urgent tasks are considered as eager.
2024-07-22 17:22:53 +08:00
ce1c268743 [Fix](load) Fix uninitiated load_type (not a valid value for type) of StreamLoadContext when operate_txn_2pc #38033 (#38187)
cherry pick from #38033
2024-07-22 17:20:03 +08:00
2e79612320 [env](compile) compile failed caused by <immintrin.h> (#37068) (#38130)
pick from master #37068
2024-07-20 14:10:31 +08:00
7819c75e55 [fix](shuffle) Fix local exchange dependency blocking (#38160)
## Proposed changes

pick #38151

<!--Describe your changes.-->
2024-07-20 00:19:47 +08:00
947d7594c8 [fix](func)fix array_with_const with larger than max_array_size (#38152)
## Proposed changes
backport: https://github.com/apache/doris/pull/37495
Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-20 00:19:00 +08:00
ffc0d6884d [Fix](load) Fix the channel leak when close wait has been cancelled #38031 (#38125)
cherry pick from #38031
2024-07-19 22:58:54 +08:00
d9fd419e47 [Fix](JsonReader) fix json with duplicate key entry may result out of bound exception (#38147)
#38146
2024-07-19 22:53:02 +08:00
22b9cc6eb2 [fix](mysql)fix mysql row buf (#38145)
## Proposed changes
backport: https://github.com/apache/doris/pull/37936
Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-19 22:18:42 +08:00
7b141ffde7 [pick]add min scan thread num for workload group's scan thread (#38123)
## Proposed changes

pick #38096
2024-07-19 18:43:05 +08:00
de2272ce48 [fix](round) fix round decimal128 overflow (#37733) (#37963)
cherry-pick #37733 to branch-2.1
2024-07-18 23:50:23 +08:00
4b31e52b24 [enhancement](runtimefilter) fix potential core in runtime filter sync filter size (#38058) (#38093)
pick #38058

## Proposed changes
IRuntimeFilter maybe deconstructed before the rpc finished, so that
could not use a raw pointer in closure. Has to use the context's shared
ptr.

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-07-18 23:11:26 +08:00
294157dafc [fix](core) fix closure use after release #38087 (#38094)
## Proposed changes
https://github.com/apache/doris/pull/38087
<!--Describe your changes.-->
2024-07-18 22:44:30 +08:00
13e9842f17 [enhancement](memtable) use shared ptr for flush token since it is shared between memtable write thread and flush thread (#38023) (#38068)
pick https://github.com/apache/doris/pull/38023
---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-07-18 19:09:40 +08:00
b7c038eb11 [fix]fix be core when migration tablet to other disk (#37712) (#38062)
![image](https://github.com/user-attachments/assets/01f67160-3ebe-41a1-ac79-a7173d14605c)
The asynchronous task reference captures a local variable
Issue Number: close #36809

Co-authored-by: z404289981 <123865460+z404289981@users.noreply.github.com>
Co-authored-by: huronghui <huronghui@huronghuideMac-mini.local>
2024-07-18 17:51:32 +08:00
8c6ff22e04 [Pick](Variant) fix heap use after free and optimize cases #37991 #37976 (#38037) 2024-07-18 16:53:09 +08:00
88d771d360 [pipeline](fix) Avoid to use a freed dependency when cancelled (#34584) (#38046)
## Proposed changes

pick #34584
<!--Describe your changes.-->
2024-07-18 15:27:10 +08:00
c30c1d2436 [branch-2.1] Picks "[opt](delete) Delete job should retry for failure that is not DELETE_INVALID_XXX #37834" (#38032)
## Proposed changes

picks https://github.com/apache/doris/pull/37834 and
https://github.com/apache/doris/pull/38043
2024-07-18 14:50:30 +08:00
0e248e3594 [fix](inverted index) Corrected the issue of no_index_match failure caused by empty data #37947 (#38002) 2024-07-18 10:04:36 +08:00
38885d4b00 [fix](load) fix memtable agg functions (#38017) (#38021)
backport #38017
2024-07-17 23:04:57 +08:00
3d5043817a Revert "[opt](serde)Optimize the filling of fixed values ​​into block columns without repeated deserialization. (#37377)" (#38007)
Reverts apache/doris#37530
Need more test, revert it temporarily
2024-07-17 21:44:25 +08:00
1875267796 [fix](routine-load) fix routine load pause when Kafka data deleted after TTL (#37288) (#37983)
pick (#37288)

When using routine load, After the data load is completed, the lag is
still a positive number:
```
  Lag: {"0":16,"1":15,"2":16,"3":16,"4":16,"5":16,"6":15,"7":16,"8":16,"9":16,"10":15,"11":16,"12":15,"13":15,"14":16,"15":16,"16":17,"17":15,"18":16,"19":15,"20":16,"21":16,"22":16,"23":16,"24":15,"25":17,"26":17,"27":16,"28":16,"29":16,"30":16,"31":17,"32":14,"33":16,"34":17,"35":16,"36":15,"37":15,"38":15,"39":16,"40":16,"41":16,"42":15,"43":15,"44":17,"45":16,"46":15,"47":15,"48":16,"49":17,"50":16,"51":15,"52":16,"53":15,"54":15,"55":17,"56":16,"57":17,"58":16,"59":16,"60":15,"61":15,"62":16,"63":16,"64":17,"65":16,"66":15,"67":16,"68":17,"69":16,"70":15,"71":17}
```
and the routing load is paused when the Kafka data reaches TTL and is
deleted, the error is `out of range`.

The reason why this happened is EOF has it offset which needed
statistics.

**note(important):**
After the bug is fixed, if you set 
```
"property.enable.partition.eof" = "false"
```
in your routine load job, it will meet the problem. For EOF has offset,
and the config is true in Doris default.
2024-07-17 13:47:26 +08:00
Pxl
db0a43bad2 [Chore](exchange) change LocalExchangeSharedState:mem_usage signed ty… (#37981)
pick from #36682
2024-07-17 13:46:51 +08:00
33b379a51d [bug](join) remove broadcast join check about shared hashtable signal (#37969)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-17 12:26:19 +08:00
b6e5281a1c [Fix](bug) fix the divide zero in local shuffle: (#37948)
## Proposed changes

cherry pick #37906 

<!--Describe your changes.-->
2024-07-17 01:03:53 +08:00
21c6b854f7 [fix](explode-json-object)fix explode json object (#37956)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-17 01:03:07 +08:00
359e50fc58 [fix](load) change tablet schema pointer to shared_ptr in memtable (#37927) (#37939)
backport #37927
2024-07-16 22:32:03 +08:00
b15ccdbe98 [Pick](Variant) pick some fix (#37922)
#37674
#37839
#37883 
#37857 
#37794
2024-07-16 21:38:47 +08:00
cc85f7b94c [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) (#37897) 2024-07-16 20:32:30 +08:00