Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
Message-Id: <201607282129.u6SLTidD027937@ip-10-146-233-104.ec2.internal>
Date: Thu, 28 Jul 2016 21:29:44 +0000
From: "Tim Armstrong (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, dev@impala.incubator.apache.org
Reply-To: tarmstrong@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-3611=3A_track_unused_Disk_IO_buffer_memory=0A?=
In-Reply-To: <gerrit.1469662442000.I8777cf76f04d34a46f53d53005412e0f1d63b5b7@gerrit.cloudera.org>
References: <gerrit.1469662442000.I8777cf76f04d34a46f53d53005412e0f1d63b5b7@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.2
archived-at: Thu, 28 Jul 2016 21:29:51 -0000

Hello Internal Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/3799

to look at the new patch set (#2).

Change subject: IMPALA-3611: track unused Disk IO buffer memory
......................................................................

IMPALA-3611: track unused Disk IO buffer memory

Track I/O buffers against separate MemTrackers. This gives us better
visibility into memory consumption from the debug webpage and from
MemTracker consumption dumps. The immediate motivation was in trying to
determine whether idle memory consumption of an impalad was caused by a
memory leak.

We add two trackers: for buffers cached in DiskIoMgr's free list,
and another for clients that don't provide a MemTracker (the only
one is BufferedBlockMgr, which will be removed at some point).

The previous code "tracked" the buffers against the process-wide
tracker, but it was a no-op outside of ASAN builds since the
process-wide tracker took its value from TCMalloc.

The test code required fixing because it assumed that buffers were
always credited against the DiskIoMgr's tracker. This only made sense
when the DiskIoMgr's tracker is the root process-wide tracker.

Fix backend test logging for disk-io-mgr-test.

Testing:
Ran exhaustive tests, ran local and cluster stress tests.

Change-Id: I8777cf76f04d34a46f53d53005412e0f1d63b5b7
---
M be/src/runtime/disk-io-mgr-scan-range.cc
M be/src/runtime/disk-io-mgr-test.cc
M be/src/runtime/disk-io-mgr.cc
M be/src/runtime/disk-io-mgr.h
M be/src/runtime/row-batch.cc
5 files changed, 238 insertions(+), 167 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/3799/2
-- 
To view, visit http://gerrit.cloudera.org:8080/3799
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8777cf76f04d34a46f53d53005412e0f1d63b5b7
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>