impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-5158: report buffer pool free memory in MemTracker
Date Thu, 25 May 2017 20:48:51 GMT
Tim Armstrong has uploaded a new patch set (#3).

Change subject: IMPALA-5158: report buffer pool free memory in MemTracker

IMPALA-5158: report buffer pool free memory in MemTracker

Clean pages and free buffers appear as untracked memory in the
MemTracker hierarchy. This was misleading since the memory is tracked
and present in the BufferPool. This change adds two MemTrackers below
the process level that accounts for this memory.

Updating global counters would be very inefficient and negate most of
the effort put into making the buffer allocator scalable. Instead the
values of the metrics are computed on demand by summing values across
all of the arena in BufferAlloctor.

The numbers reported are approximate because we do not lock any of the
BufferAllocator state and therefore don't get a consistent view of the
entire BufferAllocator at any moment in time. However they are accurate
enough to understand the general state of the system.

Also switches over ASAN to use a metric, similar to the regular TCMalloc
build so that the behaviour under ASAN diverges less.

Add some checks to unit tests to sanity-check that the numbers computed
are valid.

Manually tested by rebasing my buffer pool dev branch onto this change
and running some spilling queries. The /memz page reported:

  Process: Limit=8.35 GB Total=1005.49 MB Peak=1.01 GB
    Buffer Pool: Free Buffers: Total=391.50 MB
    Buffer Pool: Clean Pages: Total=112.00 MB
    Free Disk IO Buffers: Total=247.00 KB Peak=30.23 MB
    RequestPool=fe-eval-exprs: Total=0 Peak=4.00 KB
    RequestPool=default-pool: Total=374.30 MB Peak=416.55 MB
      Query(b9421063d13af70b:ddb9973900000000): Reservation=0 ReservationLimit=6.68 GB OtherMemory=801.09
KB Total=801.09 KB Peak=1.05 MB
      << snip >>
  Untracked Memory: Total=127.45 MB

Manually tested the ASAN change by building under ASAN, running some
queries, and inspecting the memz/ page. It reported a value of 100-200MB
untracked memory, similar to the non-ASAN build.

Change-Id: I007eb258377b33fff9f3246580d80fa551837078
M be/src/runtime/bufferpool/
M be/src/runtime/bufferpool/
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/
M be/src/runtime/bufferpool/
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/
M be/src/runtime/exec-env.h
M be/src/runtime/
M be/src/runtime/mem-tracker.h
M be/src/scheduling/
M be/src/util/
M be/src/util/memory-metrics.h
M common/thrift/metrics.json
14 files changed, 357 insertions(+), 51 deletions(-)

  git pull ssh:// refs/changes/93/6993/3
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I007eb258377b33fff9f3246580d80fa551837078
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <>
Gerrit-Reviewer: Dan Hecht <>

View raw message