Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201706152023.v5FKNooX002768@ip-10-146-233-104.ec2.internal>
Date: Thu, 15 Jun 2017 20:23:49 +0000
From: "Impala Public Jenkins (Code Review)" <gerrit@cloudera.org>
To: Tim Armstrong <tarmstrong@cloudera.com>, impala-cr@cloudera.com,
        reviews@impala.incubator.apache.org
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-5158=3A_report_buffer_pool_free_memory_in_MemTracker=0A?=
In-Reply-To: <gerrit.1495745228000.I007eb258377b33fff9f3246580d80fa551837078@gerrit.cloudera.org>
References: <gerrit.1495745228000.I007eb258377b33fff9f3246580d80fa551837078@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.7
archived-at: Thu, 15 Jun 2017 20:23:55 -0000

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-5158: report buffer pool free memory in MemTracker
......................................................................


IMPALA-5158: report buffer pool free memory in MemTracker

Clean pages and free buffers appear as untracked memory in the
MemTracker hierarchy. This was misleading since the memory is tracked
and present in the BufferPool. This change adds two MemTrackers below
the process level that accounts for this memory.

Updating global counters would be very inefficient and negate most of
the effort put into making the buffer allocator scalable. Instead the
values of the metrics are computed on demand by summing values across
all of the arena in BufferAlloctor.

The numbers reported are approximate because we do not lock any of the
BufferAllocator state and therefore don't get a consistent view of the
entire BufferAllocator at any moment in time. However they are accurate
enough to understand the general state of the system.

Also switches over ASAN to use a metric, similar to the regular TCMalloc
build so that the behaviour under ASAN diverges less.

Testing:
Add some checks to unit tests to sanity-check that the numbers computed
are valid.

Manually tested by rebasing my buffer pool dev branch onto this change
and running some spilling queries. The /memz page reported:

  Process: Limit=8.35 GB Total=1005.49 MB Peak=1.01 GB
    Buffer Pool: Free Buffers: Total=391.50 MB
    Buffer Pool: Clean Pages: Total=112.00 MB
    Free Disk IO Buffers: Total=247.00 KB Peak=30.23 MB
    RequestPool=fe-eval-exprs: Total=0 Peak=4.00 KB
    RequestPool=default-pool: Total=374.30 MB Peak=416.55 MB
      Query(b9421063d13af70b:ddb9973900000000): Reservation=0 ReservationLimit=6.68 GB OtherMemory=801.09 KB Total=801.09 KB Peak=1.05 MB
      << snip >>
  Untracked Memory: Total=127.45 MB

Manually tested the ASAN change by building under ASAN, running some
queries, and inspecting the memz/ page. It reported a value of 100-200MB
untracked memory, similar to the non-ASAN build.

Change-Id: I007eb258377b33fff9f3246580d80fa551837078
Reviewed-on: http://gerrit.cloudera.org:8080/6993
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/runtime/bufferpool/buffer-allocator-test.cc
M be/src/runtime/bufferpool/buffer-allocator.cc
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/scheduling/admission-controller.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M common/thrift/metrics.json
14 files changed, 357 insertions(+), 51 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Tim Armstrong: Looks good to me, approved


-- 
To view, visit http://gerrit.cloudera.org:8080/6993
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I007eb258377b33fff9f3246580d80fa551837078
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>