database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[MemTracker] make all MemTrackers shared (#4135)

Browse Source 
We make all MemTrackers shared, in order to show MemTracker real-time consumptions on the web.
As follows:
1. nearly all MemTracker raw ptr -> shared_ptr
2. Use CreateTracker() to create new MemTracker(in order to add itself to its parent)
3. RowBatch & MemPool still use raw ptrs of MemTracker, it's easy to ensure RowBatch & MemPool destructor exec 
     before MemTracker's destructor. So we don't change these code.
4. MemTracker can use RuntimeProfile's counter to calc consumption. So RuntimeProfile's counter need to be shared 
    too. We add a shared counter pool to store the shared counter, don't change other counters of RuntimeProfile.
Note that, this PR doesn't change the MemTracker tree structure. So there still have some orphan trackers, e.g. RowBlockV2's MemTracker. If you find some shared MemTrackers are little memory consumption & too time-consuming, you could make them be the orphan, then it's fine to use the raw ptr.
This commit is contained in:
HuangWei
2020-07-31 21:57:21 +08:00
committed by GitHub
parent f412f99511
commit 10f822eb43
209 changed files with 2137 additions and 1845 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
be/src/exec/hash_join_node.cpp
View File

@ -81,7 +81,7 @@ Status HashJoinNode::init(const TPlanNode& tnode, RuntimeState* state) {
Status HashJoinNode::prepare(RuntimeState* state) {
RETURN_IF_ERROR(ExecNode::prepare(state));
_build_pool.reset(new MemPool(mem_tracker()));
_build_pool.reset(new MemPool(mem_tracker().get()));
_build_timer =
ADD_TIMER(runtime_profile(), "BuildTime");
_push_down_timer =
@ -139,7 +139,7 @@ Status HashJoinNode::prepare(RuntimeState* state) {
_build_expr_ctxs, _probe_expr_ctxs, _build_tuple_size,
stores_nulls, _is_null_safe_eq_join, id(), mem_tracker(), 1024));
_probe_batch.reset(new RowBatch(child(0)->row_desc(), state->batch_size(), mem_tracker()));
_probe_batch.reset(new RowBatch(child(0)->row_desc(), state->batch_size(), mem_tracker().get()));
return Status::OK();
}
@ -185,7 +185,7 @@ Status HashJoinNode::construct_hash_table(RuntimeState* state) {
// The hash join node needs to keep in memory all build tuples, including the tuple
// row ptrs. The row ptrs are copied into the hash table's internal structure so they
// don't need to be stored in the _build_pool.
RowBatch build_batch(child(1)->row_desc(), state->batch_size(), mem_tracker());
RowBatch build_batch(child(1)->row_desc(), state->batch_size(), mem_tracker().get());
RETURN_IF_ERROR(child(1)->open(state));
while (true) {