Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201708052213.v75MDVhT023500@ip-10-146-233-104.ec2.internal>
Date: Sat, 5 Aug 2017 22:13:31 +0000
From: "Impala Public Jenkins (Code Review)" <gerrit@cloudera.org>
To: Tim Armstrong <tarmstrong@cloudera.com>, impala-cr@cloudera.com,
        reviews@impala.incubator.apache.org
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-4703=3A_reservation_denial_debug_action=0A?=
In-Reply-To: <gerrit.1496183940000.Ied39bb091b12156e5dc61b528c6c0cd8de3fe657@gerrit.cloudera.org>
References: <gerrit.1496183940000.Ied39bb091b12156e5dc61b528c6c0cd8de3fe657@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.7
archived-at: Sat, 05 Aug 2017 22:13:35 -0000

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-4703: reservation denial debug action
......................................................................


IMPALA-4703: reservation denial debug action

Add debug action to deny reservation increases with some probability.
This allows us to test various scenarios, particularly:
* The case when the node only gets its initial reservation and must
  run to completion without increasing its reservation.
* The case when there is some memory pressure and the node sometimes
  gets a reservation increase and sometimes doesn't.

E.g. to deny all reservation requests after an ExecNode has opened:

  set debug_action=-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0

This was applied to test_spilling. It caught a bug in the PAGG
with spilling string aggregations.

This required some minor extensions to the debug actions.
* Allow debug actions that apply to all ExecNodes if node_id is -1.
* Allow passing parameters to debug actions. The current grammar of the
  actions is not well-oriented towards extension, so I resorted to using
  @ as a new delimiter.

I also optimised ExecDebugAction() so that it is much faster in the
common case and extended --disable_mem_pools to prevent the buffer pool
from holding onto unused buffers.

Change-Id: Ied39bb091b12156e5dc61b528c6c0cd8de3fe657
Reviewed-on: http://gerrit.cloudera.org:8080/7022
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/runtime/bufferpool/buffer-allocator.cc
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/reservation-tracker.cc
M be/src/runtime/bufferpool/reservation-tracker.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/debug-options.cc
M be/src/runtime/debug-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/PlanNodes.thrift
M tests/query_test/test_spilling.py
13 files changed, 140 insertions(+), 37 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Tim Armstrong: Looks good to me, approved


-- 
To view, visit http://gerrit.cloudera.org:8080/7022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ied39bb091b12156e5dc61b528c6c0cd8de3fe657
Gerrit-PatchSet: 13
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>