Commit Graph

20 Commits

Author SHA1 Message Date
0dccc4e6e4 [cherry-pick](branch-2.1)fix http error when downloading varaint inverted index file #35668 (#36061)
pick from master[#35668](https://github.com/apache/doris/pull/35668)
2024-06-11 14:09:05 +08:00
c22f42121b [fix](compaction test) show single replica compaction status and fix test (#33076) (#34285) (#34438) 2024-05-06 21:00:34 +08:00
7e91e69eb9 [fix](compaction) fix single compaction (#33907)
* [fix](compaction)Fix single compaction to get all local versions #33849

add test and comment

* remove single replica compaction prepare input rowsets

reviesd
2024-04-19 23:30:25 +08:00
779ca464a5 [Fix](Status) Handle returned overall Status correctly (#31692)
Handle returned overall Status correctly
2024-03-09 19:44:39 +08:00
fc53c7210b [fix](chmod) change chmod to filesystem::permission to avoid race condition (#31032) 2024-02-18 11:50:16 +08:00
56648e0bd6 [Bug](security) BE SingleReplicaCompaction download_files function exists log print sensitive msg #28929 (#28930)
Co-authored-by: Rohit Satardekar <rohitrs1983@gmail.com>
2023-12-30 20:50:08 +08:00
1aa9ac4fe4 Prevent making snapshot on remote rowset in single replica compaction (#28716) 2023-12-27 23:43:43 +08:00
f374beaa4e [fix](log) regularise some BE error type and fix a load task check #28729 2023-12-25 10:45:19 +08:00
a3cd36ce60 [bug](cooldown) Fix incorrect remote rowset dir after restarting BE (#28140) 2023-12-10 00:44:01 +08:00
d767804815 [feature](merge-cloud) Decouple rowset id generator and local rowsets gc implementation (#25921) 2023-11-10 10:07:02 +08:00
9c9fc84f39 [feature](merge-cloud) Abstract BaseTablet for CloudTablet (#24929) 2023-10-18 20:29:04 +08:00
642e5cdb69 [Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly (#23395) 2023-09-29 22:38:52 +08:00
6502da8917 [bugfix](restore) add partition id into convert_rowset_ids() (#24834) 2023-09-25 20:07:24 +08:00
c8655d1dae [enhancement](compaction) print correct error messsage & avoid unnecessary stacktrace (#24758) 2023-09-22 11:58:11 +08:00
d4694167a8 [Enhancement](chore) Some Status relevant enhancement (#23072) 2023-08-21 14:14:38 +08:00
0d75a54d6c [fix](compaction) fix null pointer if single replica compaction gets rowset version from peer (#22717) 2023-08-09 20:55:24 +08:00
c409fa0f58 [Feature](Compaction)Support full compaction (#21177) 2023-07-16 13:21:15 +08:00
Pxl
ca71048f7f [Chore](status) avoid empty error msg on status (#21454)
avoid empty error msg on status
2023-07-11 13:48:16 +08:00
Pxl
7dc7ed97eb [Chore](build) remove some unused code and remove some wno (#20326)
remove some unused code about spinlock
remove some wno and fix warning
remove varadic macro usage
2023-06-05 10:48:07 +08:00
accaff1026 [Feature](compaction) wip: single replica compaction (#19237)
Currently, compaction is executed separately for each backend, and the reconstruction of the index during compaction leads to high CPU usage. To address this, we are introducing single replica compaction, where a specific primary replica is selected to perform compaction, and the remaining replicas fetch the compaction results from the primary replica.

The Backend (BE) requests replica information for all peers corresponding to a tablet from the Frontend (FE). This information includes the host where the replica is located and the replica_id. By calculating hash(replica_id), the replica with the smallest hash value is responsible for executing compaction, while the remaining replicas are responsible for fetching the compaction results from this replica.
The compaction task producer thread, before submitting a compaction task, checks whether the local replica should fetch from its peer. If it should, the task is then submitted to the single replica compaction thread pool.
When performing single replica compaction, the process begins by requesting rowset versions from the target replica. These rowset_versions are then compared with the local rowset versions. The first version that can be fetched is selected.
2023-05-30 21:12:48 +08:00