2196c534e8
[fix](group commit) Fix compatibility issues on serializing and deserializing wal file ( #32299 )
2024-03-21 14:07:24 +08:00
ff8c679851
[fix](group commit) group commit may heap-use-after-free if execute plan failed ( #31839 )
2024-03-07 16:12:23 +08:00
eea9b56f69
[fix](group commit) handle group commit create plan error ( #31757 )
2024-03-06 13:07:59 +08:00
770cdabda3
[fix](group_commit) GroupCommitBlockSink shoud not use load_block_queue when creating load task fail ( #31416 )
2024-02-29 12:38:02 +08:00
747faeed17
[Enhancement](group commit) optimize some group commit code ( #31392 )
...
This PR optimizes some of the logic related to group commit:
1. Improved the error handling when there is insufficient WAL space during import.
2. Accounted for cases where the content length is negative during import.
3. Added missing error log printing in `group_commit_mgr.cpp`.
2024-02-28 13:05:57 +08:00
9c096710fa
[Enhancement](group commit) Add bvar and log for group commit ( #31017 )
2024-02-22 13:01:32 +08:00
b5012dc55a
[Enhancement](group commit) optimize pre allocated calculation ( #30893 )
2024-02-18 11:50:17 +08:00
22cc8342d4
[Enhancement](wal) Optimize group commit manager log ( #30849 )
2024-02-16 10:12:24 +08:00
5bb8a72626
[fix](group_commit) Fix write column id twice problem on wal file ( #30935 )
2024-02-16 10:12:24 +08:00
2667e10ba2
[improve](group-commit) Modify some log ( #30842 )
2024-02-16 10:12:23 +08:00
847db2c015
[Enhancement](group commit) Add retry message for group commit load while schema changing ( #30391 )
2024-02-16 10:12:23 +08:00
82bb3ed50f
[Fix](group commit) Fix pre allocated err handling for group commit async load and add regression test #30718
2024-02-02 13:31:47 +08:00
18b5b72cd9
[improve](stack_trace) avoid print stack trace in group commit ( #30399 )
2024-01-27 09:08:29 +08:00
9c742d46a2
[fix](group commit) abort txn should use label if replay wal failed ( #30219 )
2024-01-23 10:12:35 +08:00
e1bcdc35fd
(fix)[group-commit] Fix some group commit case ( #30008 )
2024-01-16 18:46:19 +08:00
cbffdbb8bf
[bug](group_commit) fix relay wal problem on materialized-view ( #29848 )
2024-01-12 13:58:19 +08:00
c9a949130b
[Case](wal) Add wal group commit sink case with low disk space fault injection ( #29731 )
2024-01-12 11:59:52 +08:00
8c0b046ad4
[case](wal)Add wal backpressure case ( #29725 )
2024-01-12 11:59:52 +08:00
2959219b4d
[Enhancement](wal) Add fault injection case for wal back pressure ( #29689 )
2024-01-12 11:44:21 +08:00
0b731800a0
[enhancement](group_commit) refector wal manager code ( #29560 )
2024-01-07 18:54:41 +08:00
1ae23a3206
[Fix](group commit) Fix wal mem back pressure fault injection case ( #29493 )
2024-01-06 18:39:26 +08:00
eea6219920
[improvement](group commit) Add an option of enable relay wal on all regression test ( #28757 )
2024-01-05 18:27:31 +08:00
e3c9f535dc
[refactor](wal) refactor some wal code ( #29434 )
2024-01-03 14:45:57 +08:00
69524ccf98
[improve](group commit) Group commit support commit by data size ( #29428 )
2024-01-02 23:20:23 +08:00
b3f69211b7
[refactor](wal) move group commit load content length to runtime state ( #29188 )
2024-01-02 15:53:38 +08:00
706463781c
[refactor](group commit) refactor group commit wal code ( #29375 )
2024-01-02 15:52:03 +08:00
cec69f4cbd
[Enhancement](wal) Add timout for wal memory back pressure ( #29178 )
2024-01-02 11:02:17 +08:00
4cbbd25d8c
[fix](ctx) manager the lifecycle of connection context ( #29346 )
...
In FrontendService, we may create some connection context and set it as a thread local varaible.
These context should be removed from thread local after call.
Otherwise, it may be reused by other thread incorrectly.
2024-01-01 23:32:28 +08:00
03901b9a7a
[enhancement](group_commit): refector relay wal code ( #29183 )
2023-12-30 12:59:46 +08:00
9ff8bd2e9c
[Enhancement](Wal)Support dynamic wal space limit ( #27726 )
2023-12-27 11:51:32 +08:00
0af6bd6390
[fix](group-commit) check if wal need recovery is abnormal ( #28769 )
2023-12-22 11:06:11 +08:00
0070909d30
[fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception ( #28691 )
2023-12-21 20:27:33 +08:00
970e1c8475
[fix](group_commit) fix group commit cancel stuck ( #28749 )
2023-12-21 10:32:21 +08:00
66fbb22ad7
[fix](group commit) Fix some wal problems on group commit ( #28554 )
2023-12-19 09:51:03 +08:00
45b2dbab6a
[improve](group commit) Group commit support max filter ratio when rows is less than value in config ( #28139 )
2023-12-12 16:33:36 +08:00
d4f89389e3
[improve](group commit) Group commit support skip wal ( #27957 )
2023-12-11 19:38:32 +08:00
cff1de29ce
[fix](group commit) Fix group commit memory calculation ( #28242 )
2023-12-11 17:05:26 +08:00
1e5ff40e17
[refactor](group commit) remove future block ( #27720 )
...
Co-authored-by: huanghaibin <284824253@qq.com >
2023-12-11 08:41:51 +08:00
13b26ee920
[Fix](core) Fix wal space back pressure core and add regression test ( #27311 )
2023-11-27 15:10:26 +08:00
b457856bd2
[chore](be) remove bthread scanner related codes ( #27417 )
2023-11-23 15:18:49 +08:00
1cd1c58eee
[Feature](group commit) move group_commit_interval_ms from be.conf to table property ( #27116 )
2023-11-21 20:50:02 +08:00
5d548935e0
[improvement](insert) support schema change and decommission for group commit ( #26359 )
2023-11-17 21:41:38 +08:00
c0fda8c5c2
[improve](group commit) Add a swicth to wait internal group commit lo… ( #26734 )
...
* [improve](group commit) Add a swicth to make internal group commit load finish
* modify group commit tvf plan
2023-11-13 10:35:35 +08:00
b19f275714
[improvement](insert) refactor group commit insert into ( #25795 )
2023-11-03 12:02:40 +08:00
387e33fa34
[enhancement](group commit)Add group commit block queues memory back pressure ( #26045 )
2023-11-01 16:29:45 +08:00
e20cab64f4
[improvement](scan) avoid too many scanners for file scan node ( #25727 )
...
In previous, when using file scan node(eq, querying hive table), the max number of scanner for each scan node
will be the `doris_scanner_thread_pool_thread_num`(default is 48).
And if the query parallelism is N, the total number of scanner would be 48 * N, which is too many.
In this PR, I change the logic, the max number of scanner for each scan node
will be the `doris_scanner_thread_pool_thread_num / query parallelism`. So that the total number of scanners
will be up to `doris_scanner_thread_pool_thread_num`.
Reduce the number of scanner can significantly reduce the memory usage of query.
2023-10-29 17:41:31 +08:00
fded8ba824
[improvement](insert) Modify insert into table_id(xxx) to doris_inter… ( #25662 )
2023-10-20 19:31:37 +08:00
d0cd535cb9
[improvement](insert) refactor group commit stream load ( #25560 )
2023-10-20 13:27:30 +08:00
f1a5e393c7
[feature](insert) Support group commit insert use new syntax like insert into table_id(xxx) ( #25484 )
2023-10-17 09:23:09 +08:00
1073ef22f3
[fix](insert) improve group_commit related tests ( #25319 )
2023-10-12 21:19:29 +08:00
