impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Huaisi Xu (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-2.2.0_5.4.x) IMPALA-2378: check proc mem limit before preparing fragment
Date Sun, 27 Mar 2016 16:59:26 GMT
Huaisi Xu has submitted this change and it was merged.

Change subject: IMPALA-2378: check proc mem limit before preparing fragment
......................................................................


IMPALA-2378: check proc mem limit before preparing fragment

Preparing a query fragment for execution uses a significant amount of
untracked memory. Currently, if a query is admitted, it will consume
a significant amount of memory regardless of how much process memory
is available on each impalad. This is a problem since the untracked
memory is included in the Tcmalloc stats used towards thhe process
memory limit.

We saw stress tests get into a state where the vast majority of cluster
memory was taken up by this untracked memory, causing allocations of
tracked memory to fail.

This patch partially addresses the problem by checking whether the
process memory limit is exceeded before preparing a query fragment.
If there is no process memory available, fragment execution is
cancelled before any additional memory is consumed for LLVM code,
descriptors, execution nodes, and other untracked structures.
This gives already-running queries a better chance of succeeding and
does not waste resources preparing fragments that will immediately hit a
memory limit.

Testing: ran stress tests with the two TPCDS queries known to exhibit
this problem especially badly. The memory limit was not exceeded by more
than 0.1GB.

Change-Id: Ie7743834a57a044657f6fcac4623a609c2eb07dc
Reviewed-on: http://gerrit.cloudera.org:8080/1215
Reviewed-by: Dan Hecht <dhecht@cloudera.com>
Tested-by: Internal Jenkins
(cherry picked from commit b54b6930be3fb902ae8425a3e170332aa7762996)
Reviewed-on: http://gerrit.cloudera.org:8080/2639
Reviewed-by: anujphadke <aphadke@cloudera.com>
Tested-by: Huaisi Xu <hxu@cloudera.com>
---
M be/src/service/fragment-mgr.cc
1 file changed, 13 insertions(+), 0 deletions(-)

Approvals:
  Huaisi Xu: Verified
  anujphadke: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/2639
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie7743834a57a044657f6fcac4623a609c2eb07dc
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.2.0_5.4.x
Gerrit-Owner: Huaisi Xu <hxu@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hxu@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: anujphadke <aphadke@cloudera.com>

Mime
View raw message