830f250a80
[opt](query cancel) cancel query if it has pipeline task leakage #39223 ( #39537 )
...
pick #39223 with some modifications. Optimization will only be applied
to pipeline x.
2024-08-19 14:33:59 +08:00
eea3676791
[fix](group commit) fix group commit insert rpc may stuck ( #39391 ) ( #39458 )
...
pick https://github.com/apache/doris/pull/39391
2024-08-16 13:19:00 +08:00
2ec1a6a7e7
[fix](group commit) Modify group commit commit/abort txn timeout as stream load ( #39003 ) ( #39069 )
...
pick https://github.com/apache/doris/pull/39003
2024-08-08 14:36:29 +08:00
749c9f7b56
[fix](group commit) fix repaly wal check label status ( #38883 ) ( #38997 )
...
pick https://github.com/apache/doris/pull/38883
2024-08-07 22:06:59 +08:00
3abb222064
[fix](group commit) Fix test_group_commit_async_wal_msg_fault_injection case ( #35313 ) ( #38911 )
...
pick https://github.com/apache/doris/pull/35313
2024-08-06 17:57:22 +08:00
e8690b62ee
[fix](group commit) Pick add debug log show why group commit not work; delete wal when replay success ( #38611 ) ( #38659 )
...
Pick https://github.com/apache/doris/pull/38611
2024-08-01 16:59:54 +08:00
91f0301b43
[fix](group commit) Pick some group commit pr ( #38320 )
...
Pick https://github.com/apache/doris/pull/38292 ,
https://github.com/apache/doris/pull/34021 ,
https://github.com/apache/doris/pull/38228 , some modify of
https://github.com/apache/doris/pull/37260 , some modify of
https://github.com/apache/doris/pull/37595
2024-07-25 17:32:44 +08:00
792bd7c74a
[cherry-pick](branch-2.1) Pick "[Enhancement](group commit)Optimize be select for group commit #35558 " ( #37830 )
...
Pick #35558
2024-07-24 09:21:07 +08:00
9556c07a16
[mac](compile) fix compile error on mac ( #37726 )
2024-07-15 10:19:42 +08:00
1e3ab0ff8c
[fix](group commit) Pick make group commit cancel in time ( #36249 ) ( #37404 )
...
pick https://github.com/apache/doris/pull/36249/
2024-07-09 09:25:11 +08:00
1a25270918
[fix](group commit) Pick Fix the incorrect group commit count in log; fix the core in get_first_block ( #36408 ) ( #37405 )
...
Pick https://github.com/apache/doris/pull/36408/
2024-07-09 09:24:43 +08:00
70f46c12b3
[improve](group commit) Pick Modify group commit case and modify cancel status ( #35995 ) ( #37398 )
...
Pick https://github.com/apache/doris/pull/35995
2024-07-08 10:27:08 +08:00
7d423b3a6a
[chery-pick](branch-2.1) Pick "[Fix](group commit) Fix group commit block queue mem estimate fault" ( #37379 )
...
Pick [Fix](group commit) Fix group commit block queue mem estimate faule
#35314
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
**Problem:** When `group commit=async_mode` and NULL data is imported
into a `variant` type column, it causes incorrect memory statistics for
group commit backpressure, leading to a stuck issue. **Cause:** In group
commit mode, blocks are first added to a queue in batches using `add
block`, and then blocks are retrieved from the queue using `get block`.
To track memory usage during backpressure, we add the block size to the
memory statistics during `add block` and subtract the block size from
the memory statistics during `get block`. However, for `variant` types,
during the `add block` write to WAL, serialization occurs, which can
merge types (e.g., merging `int` and `bigint` into `bigint`), thereby
changing the block size. This results in a discrepancy between the block
size during `get block` and `add block`, causing memory statistics to
overflow.
**Solution:** Record the block size at the time of `add block` and use
this recorded size during `get block` instead of the actual block size.
This ensures consistency in the memory addition and subtraction.
## Further comments
If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org ](mailto:dev@doris.apache.org ) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-07 18:27:49 +08:00
8b061c7055
[Enhancement](group commit) Add fault injection case for group commit
2024-04-19 15:03:06 +08:00
cf7595d423
[opt](memory) Optimize mem tracker accuracy ( #32039 ) ( #33140 )
2024-04-10 11:42:19 +08:00
2196c534e8
[fix](group commit) Fix compatibility issues on serializing and deserializing wal file ( #32299 )
2024-03-21 14:07:24 +08:00
ff8c679851
[fix](group commit) group commit may heap-use-after-free if execute plan failed ( #31839 )
2024-03-07 16:12:23 +08:00
eea9b56f69
[fix](group commit) handle group commit create plan error ( #31757 )
2024-03-06 13:07:59 +08:00
770cdabda3
[fix](group_commit) GroupCommitBlockSink shoud not use load_block_queue when creating load task fail ( #31416 )
2024-02-29 12:38:02 +08:00
747faeed17
[Enhancement](group commit) optimize some group commit code ( #31392 )
...
This PR optimizes some of the logic related to group commit:
1. Improved the error handling when there is insufficient WAL space during import.
2. Accounted for cases where the content length is negative during import.
3. Added missing error log printing in `group_commit_mgr.cpp`.
2024-02-28 13:05:57 +08:00
9c096710fa
[Enhancement](group commit) Add bvar and log for group commit ( #31017 )
2024-02-22 13:01:32 +08:00
b5012dc55a
[Enhancement](group commit) optimize pre allocated calculation ( #30893 )
2024-02-18 11:50:17 +08:00
22cc8342d4
[Enhancement](wal) Optimize group commit manager log ( #30849 )
2024-02-16 10:12:24 +08:00
5bb8a72626
[fix](group_commit) Fix write column id twice problem on wal file ( #30935 )
2024-02-16 10:12:24 +08:00
2667e10ba2
[improve](group-commit) Modify some log ( #30842 )
2024-02-16 10:12:23 +08:00
847db2c015
[Enhancement](group commit) Add retry message for group commit load while schema changing ( #30391 )
2024-02-16 10:12:23 +08:00
82bb3ed50f
[Fix](group commit) Fix pre allocated err handling for group commit async load and add regression test #30718
2024-02-02 13:31:47 +08:00
18b5b72cd9
[improve](stack_trace) avoid print stack trace in group commit ( #30399 )
2024-01-27 09:08:29 +08:00
9c742d46a2
[fix](group commit) abort txn should use label if replay wal failed ( #30219 )
2024-01-23 10:12:35 +08:00
e1bcdc35fd
(fix)[group-commit] Fix some group commit case ( #30008 )
2024-01-16 18:46:19 +08:00
cbffdbb8bf
[bug](group_commit) fix relay wal problem on materialized-view ( #29848 )
2024-01-12 13:58:19 +08:00
c9a949130b
[Case](wal) Add wal group commit sink case with low disk space fault injection ( #29731 )
2024-01-12 11:59:52 +08:00
8c0b046ad4
[case](wal)Add wal backpressure case ( #29725 )
2024-01-12 11:59:52 +08:00
2959219b4d
[Enhancement](wal) Add fault injection case for wal back pressure ( #29689 )
2024-01-12 11:44:21 +08:00
0b731800a0
[enhancement](group_commit) refector wal manager code ( #29560 )
2024-01-07 18:54:41 +08:00
1ae23a3206
[Fix](group commit) Fix wal mem back pressure fault injection case ( #29493 )
2024-01-06 18:39:26 +08:00
eea6219920
[improvement](group commit) Add an option of enable relay wal on all regression test ( #28757 )
2024-01-05 18:27:31 +08:00
e3c9f535dc
[refactor](wal) refactor some wal code ( #29434 )
2024-01-03 14:45:57 +08:00
69524ccf98
[improve](group commit) Group commit support commit by data size ( #29428 )
2024-01-02 23:20:23 +08:00
b3f69211b7
[refactor](wal) move group commit load content length to runtime state ( #29188 )
2024-01-02 15:53:38 +08:00
706463781c
[refactor](group commit) refactor group commit wal code ( #29375 )
2024-01-02 15:52:03 +08:00
cec69f4cbd
[Enhancement](wal) Add timout for wal memory back pressure ( #29178 )
2024-01-02 11:02:17 +08:00
4cbbd25d8c
[fix](ctx) manager the lifecycle of connection context ( #29346 )
...
In FrontendService, we may create some connection context and set it as a thread local varaible.
These context should be removed from thread local after call.
Otherwise, it may be reused by other thread incorrectly.
2024-01-01 23:32:28 +08:00
03901b9a7a
[enhancement](group_commit): refector relay wal code ( #29183 )
2023-12-30 12:59:46 +08:00
9ff8bd2e9c
[Enhancement](Wal)Support dynamic wal space limit ( #27726 )
2023-12-27 11:51:32 +08:00
0af6bd6390
[fix](group-commit) check if wal need recovery is abnormal ( #28769 )
2023-12-22 11:06:11 +08:00
0070909d30
[fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception ( #28691 )
2023-12-21 20:27:33 +08:00
970e1c8475
[fix](group_commit) fix group commit cancel stuck ( #28749 )
2023-12-21 10:32:21 +08:00
66fbb22ad7
[fix](group commit) Fix some wal problems on group commit ( #28554 )
2023-12-19 09:51:03 +08:00
45b2dbab6a
[improve](group commit) Group commit support max filter ratio when rows is less than value in config ( #28139 )
2023-12-12 16:33:36 +08:00
