Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201708012116.v71LGRKZ013606@ip-10-146-233-104.ec2.internal>
Date: Tue, 1 Aug 2017 21:16:26 +0000
From: "Tim Armstrong (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org
CC: Dan Hecht <dhecht@cloudera.com>
Reply-To: tarmstrong@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-5158=2CIMPALA-5236=3A_account_for_unused_buffer_pool_reservations=0A?=
In-Reply-To: <gerrit.1499462524000.Idb1fa3110dc893321f9f4e8ced6b7ede12194dad@gerrit.cloudera.org>
References: <gerrit.1499462524000.Idb1fa3110dc893321f9f4e8ced6b7ede12194dad@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.7
archived-at: Tue, 01 Aug 2017 21:16:32 -0000

Tim Armstrong has uploaded a new patch set (#4).

Change subject: IMPALA-5158,IMPALA-5236: account for unused buffer pool reservations
......................................................................

IMPALA-5158,IMPALA-5236: account for unused buffer pool reservations

We were missing accounting for this, since it is part of the expected
difference between query and process memory consumption. The identity
that applies is is:

  buffers allocated from system =
      reservation + cached buffers - unused reservation

Where "cached buffers" includes free buffers and buffers attached to
clean pages. The reservation is accounted against queries and "buffers
allocated from system" is accounted against the process MemTracker.

Reporting this in a direct way required adding a concept of a MemTracker
with negative consumption, which fortunately did not require any major
changes to the MemTracker code.

Example output when applied to buffer pool branch:

  Process: Limit=8.35 GB Total=579.18 MB Peak=590.41 MB
    Buffer Pool: Free Buffers: Total=268.25 MB
    Buffer Pool: Clean Pages: Total=172.25 MB
    Buffer Pool: Unused Reservation: Total=-8.25 MB
    Free Disk IO Buffers: Total=21.98 MB Peak=21.98 MB
    RequestPool=default-pool: Total=12.07 MB Peak=71.58 MB
      ... <snip> ...
    RequestPool=fe-eval-exprs: Total=0 Peak=4.00 KB
    Untracked Memory: Total=112.88 MB

Testing:
Added a basic test for MemTrackers with negative metrics.

Change-Id: Idb1fa3110dc893321f9f4e8ced6b7ede12194dad
---
M be/src/runtime/exec-env.cc
M be/src/runtime/mem-tracker-test.cc
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M be/src/util/pretty-printer-test.cc
M be/src/util/pretty-printer.h
M common/thrift/metrics.json
9 files changed, 76 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/7380/4
-- 
To view, visit http://gerrit.cloudera.org:8080/7380
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb1fa3110dc893321f9f4e8ced6b7ede12194dad
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>