impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-4748: crash in TmpFileMgr when hitting process mem limit
Date Thu, 12 Jan 2017 02:08:05 GMT
Tim Armstrong has uploaded a new change for review.

Change subject: IMPALA-4748: crash in TmpFileMgr when hitting process mem limit

IMPALA-4748: crash in TmpFileMgr when hitting process mem limit

The bug is that FileGroup didn't correctly handle its 'io_ctx_'
being asynchronously cancelled: it left the WriteHandle in an
invalid state. This could happen when the process memory limit
was exceeded.

I fixed this in two ways (either of which would be sufficient
to avoid this exact crash):
* Fix the error handling in TmpFileMgr so that things are left
  in a valid state on the error path.
* Stop DiskIoMgr from asynchronously cancelling I/O contexts
  with no associated MemTracker. The mem_limit check and error
  propagation is necessary when DiskIoMgr will allocate memory
  on behalf of the client, but is not necessary when it is not
  allocating memory for the client - it just added a redundant
  error propagation mechanism.

This scenario should no longer be possible for BufferedBlockMgr
since DiskIoMgr won't cancel its I/O context, since it has no
associated MemTracker. However, to test that errors on this path
are correctly handled, I added a simple unit test to TmpFileMgr
that forces cancellation of the I/O context.

Change-Id: Ib0a624212bc17f7824e6d14ad143c0d5894206f8
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/tmp-file-mgr.h
4 files changed, 48 insertions(+), 11 deletions(-)

  git pull ssh:// refs/changes/83/5683/2
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib0a624212bc17f7824e6d14ad143c0d5894206f8
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <>

View raw message