impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-3671: Add query option to limit scratch space usage
Date Fri, 23 Sep 2016 21:44:58 GMT
Tim Armstrong has uploaded a new patch set (#2).

Change subject: IMPALA-3671: Add query option to limit scratch space usage

IMPALA-3671: Add query option to limit scratch space usage

Currently we can only disable spilling via a startup option which means
we need to restart the cluster for this.
This patch adds a new query option 'SCRATCH_LIMIT' that limits the amount of
scratch directory space that can be used. This would be useful to prevent
runaway queries or to prevent queries from spilling when that is not desired.
This also adds a 'ScratchSpace' counter to the runtime profile of the
BlockMgr that keeps track of the scratch space allocated.

Valid values for the SCRATCH_LIMIT query option are:
- unspecified or a limit of -1 means no limit
- a limit of 0 (zero) means spilling is disabled
- an int (= number of bytes)
- a float followed by "M" (MB) or "G" (GB)

A new test file "" was added for testing functionality.

Change-Id: Ibf8842626ded1345b632a0ccdb9a580e6a0ad470
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/buffered-block-mgr.h
M be/src/runtime/
M be/src/runtime/test-env.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/tmp-file-mgr.h
M be/src/service/
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/
A tests/query_test/
14 files changed, 407 insertions(+), 104 deletions(-)

  git pull ssh:// refs/changes/97/4497/2
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibf8842626ded1345b632a0ccdb9a580e6a0ad470
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <>
Gerrit-Reviewer: Bikramjeet Vig <>
Gerrit-Reviewer: Dan Hecht <>
Gerrit-Reviewer: Tim Armstrong <>

View raw message