database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[fix](memtracker) Introduce orphan mem tracker to verify memory tracking accuracy (#12794)

Browse Source 
The mem hook consumes the orphan tracker by default. If the thread does not attach other trackers, by default all consumption will be passed to the process tracker through the orphan tracker.

In real time, consumption of all other trackers + orphan tracker consumption = process tracker consumption.

Ideally, all threads are expected to attach to the specified tracker, so that "all memory has its own ownership", and the consumption of the orphan mem tracker is close to 0, but greater than 0.
This commit is contained in:
Xinyi Zou
2022-09-21 15:47:10 +08:00
committed by GitHub
parent 8f4bb0f804
commit b41eaa5ac0
13 changed files with 97 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
be/src/runtime/runtime_state.cpp
View File

@ -218,14 +218,13 @@ Status RuntimeState::init(const TUniqueId& fragment_instance_id, const TQueryOpt
Status RuntimeState::init_mem_trackers(const TUniqueId& query_id) {
bool has_query_mem_tracker = _query_options.__isset.mem_limit && (_query_options.mem_limit > 0);
int64_t bytes_limit = has_query_mem_tracker ? _query_options.mem_limit : -1;
if (bytes_limit > ExecEnv::GetInstance()->process_mem_tracker_raw()->limit()) {
if (bytes_limit > ExecEnv::GetInstance()->process_mem_tracker()->limit()) {
VLOG_NOTICE << "Query memory limit " << PrettyPrinter::print(bytes_limit, TUnit::BYTES)
<< " exceeds process memory limit of "
<< PrettyPrinter::print(
ExecEnv::GetInstance()->process_mem_tracker_raw()->limit(),
TUnit::BYTES)
<< PrettyPrinter::print(ExecEnv::GetInstance()->process_mem_tracker()->limit(),
TUnit::BYTES)
<< ". Using process memory limit instead";
bytes_limit = ExecEnv::GetInstance()->process_mem_tracker_raw()->limit();
bytes_limit = ExecEnv::GetInstance()->process_mem_tracker()->limit();
}
auto mem_tracker_counter = ADD_COUNTER(&_profile, "MemoryLimit", TUnit::BYTES);
mem_tracker_counter->set(bytes_limit);