impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4748: crash in TmpFileMgr when hitting process mem limit
Date Thu, 12 Jan 2017 02:08:05 GMT
Tim Armstrong has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/5683

Change subject: IMPALA-4748: crash in TmpFileMgr when hitting process mem limit
......................................................................

IMPALA-4748: crash in TmpFileMgr when hitting process mem limit

The bug is that FileGroup didn't correctly handle its 'io_ctx_'
being asynchronously cancelled: it left the WriteHandle in an
invalid state. This could happen when the process memory limit
was exceeded.

I fixed this in two ways (either of which would be sufficient
to avoid this exact crash):
* Fix the error handling in TmpFileMgr so that things are left
  in a valid state on the error path.
* Stop DiskIoMgr from asynchronously cancelling I/O contexts
  with no associated MemTracker. The mem_limit check and error
  propagation is necessary when DiskIoMgr will allocate memory
  on behalf of the client, but is not necessary when it is not
  allocating memory for the client - it just added a redundant
  error propagation mechanism.

Testing:
This scenario should no longer be possible for BufferedBlockMgr
since DiskIoMgr won't cancel its I/O context, since it has no
associated MemTracker. However, to test that errors on this path
are correctly handled, I added a simple unit test to TmpFileMgr
that forces cancellation of the I/O context.

Change-Id: Ib0a624212bc17f7824e6d14ad143c0d5894206f8
---
M be/src/runtime/disk-io-mgr.cc
M be/src/runtime/tmp-file-mgr-test.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
4 files changed, 48 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/5683/2
-- 
To view, visit http://gerrit.cloudera.org:8080/5683
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib0a624212bc17f7824e6d14ad143c0d5894206f8
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>

Mime
View raw message