fed632bf4a
[fix](move-memtable) check segment num when closing each tablet ( #36753 ) ( #37536 )
...
cherry-pick #36753 and #37660
2024-07-11 20:33:44 +08:00
bd47d5a681
[branch-2.1](auto-partition) Fix auto partition load failure in multi replica ( #36586 )
...
this pr
1. picked #35630 , which was reverted #36098 before.
2. picked #36344 from master
these two pr fixed existing bug about auto partition load.
---------
Co-authored-by: Kaijie Chen <ckj@apache.org >
2024-06-20 17:51:18 +08:00
6bb670ab38
[metrics](bvar) add bvar for load stream and file writer count ( #36300 ) ( #36336 )
...
cherry-pick #36300
2024-06-16 10:14:59 +08:00
3b23eee37c
Revert "[fix](auto-partition) fix auto partition load lost data in multi sender ( #35287 )" ( #36098 )
...
Reverts apache/doris#35630 because it brought some more damaging bugs.
we will fix it and merge in next version
2024-06-11 17:11:42 +08:00
c2fc485327
[fix](auto-partition) fix auto partition load lost data in multi sender ( #35287 ) ( #35630 )
...
## Proposed changes
Change `use_cnt` mechanism for incremental (auto partition) channels and
streams, it's now dynamically counted.
Use `close_wait()` of regular partitions as a synchronize point to make
sure all sinks are in close phase before closing any incremental (auto
partition) channels and streams.
Add dummy (fake) partition and tablet if there is no regular partition
in the auto partition table.
Backport #35287
Co-authored-by: zhaochangle <zhaochangle@selectdb.com >
2024-05-31 10:27:03 +08:00
fefbde8927
[log](move-memtable) improve logs in vtablet_writer_v2 and load_stream ( #33103 )
2024-04-12 15:09:25 +08:00
cf7595d423
[opt](memory) Optimize mem tracker accuracy ( #32039 ) ( #33140 )
2024-04-10 11:42:19 +08:00
49d17f2be2
[fix](move-memtable) fix potential duplicate of TabletStream profile ( #30397 )
2024-01-30 15:30:14 +08:00
1a51d04cb8
[fix](move-memtable) fix schema use-after-free in delta writer v2 ( #30254 )
2024-01-24 10:00:25 +08:00
fd6ec97731
[fix](injection) don't disturb CLOSE_LOAD message in LoadStream ( #30097 )
2024-01-19 15:48:56 +08:00
41875a0bf5
[fix](move-memtable) check segment id in add_segment ( #29898 )
2024-01-16 18:30:23 +08:00
bfd23e30f6
[improve](load) handle EAGAIN in load stream ( #29437 )
2024-01-04 23:02:11 +08:00
96acef908a
[fix](move-memtable) check eos when close stream ( #29547 )
2024-01-04 22:56:52 +08:00
329d57fdd7
[regression](move-memtable) test LoadStream on_idle_timeout ( #29354 )
...
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2024-01-03 14:07:51 +08:00
1e8bb75182
[improve](move-memtable) add log on idle timeout ( #29438 )
2024-01-03 11:26:26 +08:00
243b075eef
[improve](move-memtable) report failed tablets with status ( #29351 )
2024-01-02 20:24:53 +08:00
cc40cdf2d4
[improve](load) don't limit add_segment running count in load stream ( #29332 )
...
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com >
2023-12-31 16:47:46 +08:00
e62857e5a4
[improve](move-memtable) return error when back pressure cost too much time ( #29327 )
2023-12-30 01:31:05 +08:00
b31494b18c
[test](regression) add fault injection cases for LoadStream ( #29101 )
...
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2023-12-28 16:16:26 +08:00
4a60d01dc7
[improve](move-memtable) increase load_stream_flush_token_max_tasks ( #29011 )
2023-12-26 17:08:49 +08:00
3b830f89a7
[improve](move-memtable) avoid using heavy work pool during append data ( #28745 )
2023-12-22 22:51:30 +08:00
18c9ebce95
[improve](move-memtable) tweak load stream flush token num and max tasks ( #28884 )
2023-12-22 20:08:47 +08:00
db523dafcb
[improve](move-memtable) limit task num in load stream flush token ( #28748 )
2023-12-21 12:19:58 +08:00
4e86f9bab5
[improve](move-memtable) include and check offset when append data ( #28159 )
2023-12-09 16:21:36 +08:00
9c63dfd692
[improve](move-memtable) support variant in move-memtable ( #28084 )
...
* [improve](move-memtable) support variant in move-memtable
Pass the flush schema to the destination end of the sink.
2023-12-07 16:41:38 +08:00
c5f00fa746
[fix](move-memtable) temporary disable load stream profile ( #28029 )
2023-12-07 12:05:29 +08:00
75d0beb8cc
[fix](move-memtable) only report load stream profile in the end ( #27983 )
2023-12-05 11:30:54 +08:00
b457856bd2
[chore](be) remove bthread scanner related codes ( #27417 )
2023-11-23 15:18:49 +08:00
b79f5d77f1
[improve](move-memtable) improve logging messages ( #27443 )
2023-11-23 11:46:29 +08:00
f6a9914bc7
[feature](move-memtable) support auto partition in sink v2 ( #26914 )
2023-11-14 11:39:44 +08:00
58bf79f79e
[fix](move-memtable) pass load stream num to backends ( #26198 )
2023-11-08 16:16:33 +08:00
4995ca8fba
[fix](move-memtable) ensure segment is flushed before add segment ( #26522 )
2023-11-07 22:42:16 +08:00
32b36d3c9c
[refactor](move-memtable) rename proto OpenStreamSink to OpenLoadStream ( #26527 )
2023-11-07 22:41:20 +08:00
69c3e08699
[improve](load) set load id in signal handler ( #25813 )
2023-10-24 21:58:39 +08:00
b9ddcbf729
[feature](merge-cloud) Rewrite code related to IOContext ( #24269 )
2023-09-15 19:57:58 +08:00
a542f107db
[feature](move-memtable) buffer messages in load stream stub ( #23721 )
2023-09-02 13:42:34 +08:00
14310ad30b
[improvement](move-memtable) wait StreamClose from remote ( #23605 )
...
* [fix](move-memtable) wait StreamClose from remote
2023-08-30 18:03:36 +08:00
98d0a2f6c1
[feature](move-memtable)[3/7] add load stream manager and rpc service ( #23415 )
...
Co-authored-by: zhengyu <freeman.zhang1992@gmail.com >
Co-authored-by: Yongqiang YANG <dataroaring@gmail.com >
Co-authored-by: laihui <1353307710@qq.com >
2023-08-25 00:08:04 +08:00
