## Design
### Trigger
Every time when a rowset writer produces more than N (e.g. 10) segments, we trigger segment compaction. Note that only one segment compaction job for a single rowset at a time to ensure no recursing/queuing nightmare.
### Target Selection
We collect segments during every trigger. We skip big segments whose row num > M (e.g. 10000) coz we get little benefits from compacting them comparing our effort. Hence, we only pick the 'Longest Consecutive Small" segment group to do actual compaction.
### Compaction Process
A new thread pool is introduced to help do the job. We submit the above-mentioned 'Longest Consecutive Small" segment group to the pool. Then the worker thread does the followings:
- build a MergeIterator from the target segments
- create a new segment writer
- for each block readed from MergeIterator, the Writer append it
### SegID handling
SegID must remain consecutive after segment compaction.
If a rowset has small segments named seg_0, seg_1, seg_2, seg_3 and a big segment seg_4:
- we create a segment named "seg_0-3" to save compacted data for seg_0, seg_1, seg_2 and seg_3
- delete seg_0, seg_1, seg_2 and seg_3
- rename seg_0-3 to seg_0
- rename seg_4 to seg_1
It is worth noticing that we should wait inflight segment compaction tasks to finish before building rowset meta and committing this txn.
554f566217