impala-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tarmstr...@apache.org
Subject [1/8] impala git commit: IMPALA-5848: Account for TCMalloc overhead in MemTracker
Date Thu, 14 Dec 2017 17:48:38 GMT
Repository: impala
Updated Branches:
  refs/heads/master 0936e3296 -> 3cbbaf3b3


IMPALA-5848: Account for TCMalloc overhead in MemTracker

This patch adds a new MemTracker under the Process MemTracker called
"TCMalloc Overhead" which accounts for different cache freelists
maintained by TCMalloc. This added accounting also helps bring down
the amount of untracked memory.
An example dump of the Process MemTracker now looks like:
Process: Limit=8.34 GB Total=119.10 MB Peak=119.10 MB
  Buffer Pool: Free Buffers: Total=0
  Buffer Pool: Clean Pages: Total=0
  Buffer Pool: Unused Reservation: Total=0
  TCMalloc Overhead: Total=11.42 MB
  Untracked Memory: Total=107.69 MB

Testing:
Tested manually by checking the memz webpage.

Change-Id: I602e9d5e8e8d7470dcfe4addde3265057c16263a
Reviewed-on: http://gerrit.cloudera.org:8080/8782
Reviewed-by: Bikramjeet Vig <bikramjeet.vig@cloudera.com>
Tested-by: Impala Public Jenkins


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/9a6e5fa9
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/9a6e5fa9
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/9a6e5fa9

Branch: refs/heads/master
Commit: 9a6e5fa99650c4d916aa577742a7758af53f14ee
Parents: 0936e32
Author: Bikramjeet Vig <bikramjeet.vig@cloudera.com>
Authored: Wed Dec 6 14:55:40 2017 -0800
Committer: Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org>
Committed: Wed Dec 13 02:05:21 2017 +0000

----------------------------------------------------------------------
 be/src/runtime/exec-env.cc | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/9a6e5fa9/be/src/runtime/exec-env.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/exec-env.cc b/be/src/runtime/exec-env.cc
index 6fb572c..2df696e 100644
--- a/be/src/runtime/exec-env.cc
+++ b/be/src/runtime/exec-env.cc
@@ -336,6 +336,21 @@ Status ExecEnv::Init() {
   if (!aggressive_decommit_enabled) {
     return Status("TCMalloc aggressive decommit is required but is disabled.");
   }
+  // A MemTracker for TCMalloc overhead which is the difference between the physical bytes
+  // reserved (TcmallocMetric::PHYSICAL_BYTES_RESERVED) and the bytes in use
+  // (TcmallocMetrics::BYTES_IN_USE). This overhead accounts for all the cached freelists
+  // used by TCMalloc.
+  IntGauge* negated_bytes_in_use = obj_pool_->Add(new NegatedGauge<int64_t>(
+      MakeTMetricDef("negated_tcmalloc_bytes_in_use", TMetricKind::GAUGE, TUnit::BYTES),
+      TcmallocMetric::BYTES_IN_USE));
+  vector<IntGauge*> overhead_metrics;
+  overhead_metrics.push_back(negated_bytes_in_use);
+  overhead_metrics.push_back(TcmallocMetric::PHYSICAL_BYTES_RESERVED);
+  SumGauge<int64_t>* tcmalloc_overhead = obj_pool_->Add(new SumGauge<int64_t>(
+      MakeTMetricDef("tcmalloc_overhead", TMetricKind::GAUGE, TUnit::BYTES),
+      overhead_metrics));
+  obj_pool_->Add(
+      new MemTracker(tcmalloc_overhead, -1, "TCMalloc Overhead", mem_tracker_.get()));
 #endif
   mem_tracker_->RegisterMetrics(metrics_.get(), "mem-tracker.process");
 


Mime
View raw message