impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Huaisi Xu (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-2.2.0_5.4.x) IMPALA-2378: check proc mem limit before preparing fragment
Date Fri, 25 Mar 2016 23:47:14 GMT
Hello Internal Jenkins, Dan Hecht,

I'd like you to do a code review.  Please visit

    http://gerrit.cloudera.org:8080/2639

to review the following change.

Change subject: IMPALA-2378: check proc mem limit before preparing fragment
......................................................................

IMPALA-2378: check proc mem limit before preparing fragment

Preparing a query fragment for execution uses a significant amount of
untracked memory. Currently, if a query is admitted, it will consume
a significant amount of memory regardless of how much process memory
is available on each impalad. This is a problem since the untracked
memory is included in the Tcmalloc stats used towards thhe process
memory limit.

We saw stress tests get into a state where the vast majority of cluster
memory was taken up by this untracked memory, causing allocations of
tracked memory to fail.

This patch partially addresses the problem by checking whether the
process memory limit is exceeded before preparing a query fragment.
If there is no process memory available, fragment execution is
cancelled before any additional memory is consumed for LLVM code,
descriptors, execution nodes, and other untracked structures.
This gives already-running queries a better chance of succeeding and
does not waste resources preparing fragments that will immediately hit a
memory limit.

Testing: ran stress tests with the two TPCDS queries known to exhibit
this problem especially badly. The memory limit was not exceeded by more
than 0.1GB.

Change-Id: Ie7743834a57a044657f6fcac4623a609c2eb07dc
Reviewed-on: http://gerrit.cloudera.org:8080/1215
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
(cherry picked from commit b54b6930be3fb902ae8425a3e170332aa7762996)
---
M be/src/service/fragment-mgr.cc
1 file changed, 13 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/39/2639/1
-- 
To view, visit http://gerrit.cloudera.org:8080/2639
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie7743834a57a044657f6fcac4623a609c2eb07dc
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.2.0_5.4.x
Gerrit-Owner: Huaisi Xu <hxu@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>

Mime
View raw message