database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[compaction] Update cumulative point calculate algorithm (#3690)

Browse Source 
Current cumulative point calculate algorithm may skip singleton rowset when the rowset has only one segment and with NONOVERLAPPING flag. When a tablet is new created and cumulate many singleton rowsets, cumulative point will be calculated as the max version + 1, and then cumulative compaction couldn't pick any rowsets and compaction failed, and
will lead the next base compaction on this tablet with all rowsets, which can also cause memory consume problem, suppose there are thousands of rowsets.
    All singleton rowsets must be newly wrote by delta writer and hasn't
do any compaction, we should place cumulative point before any of these rowsets.
This commit is contained in:
Yingchun Lai
2020-05-30 10:34:53 +08:00
committed by GitHub
parent 7524c5ef63
commit 43d25afa2c
1 changed files with 2 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
be/src/olap/tablet.cpp
View File

@ -674,16 +674,8 @@ void Tablet::calculate_cumulative_point() {
// There is a hole, do not continue
break;
}
// break the loop if segments in this rowset is overlapping, or overlap flag is NOT NONOVERLAPPING
// even if is_segments_overlapping() return false, the overlap flag may be OVERLAPPING.
// eg: tablet with versions(rowsets):
// [0-1] NONOVERLAPPING
// [2-2] OVERLAPPING
// [2-2]'s overlap flag is OVERLAPPING because it is newly written by the delta writer.
// but is has only one segment, so is_segments_overlapping() will return false.
// but we should not continue increasing the cumulative point, because we need
// the compaction process to change the overlap flag from OVERLAPPING to NONOVERLAPPING.
if (rs->is_segments_overlapping() || rs->segments_overlap() != NONOVERLAPPING) {
// break the loop if segments in this rowset is overlapping, or is a singleton.
if (rs->is_segments_overlapping() || rs->is_singleton_delta()) {
_cumulative_point = rs->version().first;
break;
}