impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-2800: make memory maintenance less aggressive
Date Fri, 14 Apr 2017 19:27:10 GMT
Tim Armstrong has uploaded a new patch set (#4).

Change subject: IMPALA-2800: make memory maintenance less aggressive
......................................................................

IMPALA-2800: make memory maintenance less aggressive

Memory maintenance isn't necessary for correctness - the MemTracker GC
functions are sufficient to free up memory before a mem limit would be
hit. So the main purpose of memory maintenance is to avoid holding onto
unneeded memory while the system is idle or under light load. It can
also help to redistribute memory between arenas in the buffer pool if
they have become imbalanced. Neither of these purposes require the
maintenance to be very aggressive.

* Simplify the heuristic for freeing and update the comments to reflect
  the current reality.
* Limit the rate at which TCMalloc releases memory to avoid it holding
  the global heap lock for a long time.
* Increase the memory maintenance interval - 1s is too aggressive and
  can free memory that will be imminently reused by a running query.

Testing:
Tried to reproduce the original problem in a couple of ways:
* Ran a big aggregation locally and cancelled it
* Looked at memz/ of some live clusters (production and stress test).
In all cases "Bytes in page heap freelist" was 0.

This confirms that the original problem was solved by the gperftools
2.4 upgrade.

Ran a couple of stress tests locally and checked that "Bytes in page
heap freelist" was 0 after the change and that memory consumption
was generally sensible.

Change-Id: I0f822b294ab253d6f2828fc52f353aecaaf9b701
---
M be/src/common/init.cc
M be/src/util/memory-metrics.h
2 files changed, 27 insertions(+), 28 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/6626/4
-- 
To view, visit http://gerrit.cloudera.org:8080/6626
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0f822b294ab253d6f2828fc52f353aecaaf9b701
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: anujphadke <aphadke@cloudera.com>

Mime
View raw message