Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201706152223.v5FMNwCM007153@ip-10-146-233-104.ec2.internal>
Date: Thu, 15 Jun 2017 22:23:58 +0000
From: "Tim Armstrong (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org
Reply-To: tarmstrong@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-5497=3A_spilling_hash_joins_that_output_build_rows_hit_OOM=0A?=
In-Reply-To: <gerrit.1497458056000.I075388d348499c5692d044ac1bc38dd8dd0b10c7@gerrit.cloudera.org>
References: <gerrit.1497458056000.I075388d348499c5692d044ac1bc38dd8dd0b10c7@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.7
archived-at: Thu, 15 Jun 2017 22:24:05 -0000

Tim Armstrong has uploaded a new patch set (#2).

Change subject: IMPALA-5497: spilling hash joins that output build rows hit OOM
......................................................................

IMPALA-5497: spilling hash joins that output build rows hit OOM

The bug is that the join tried to bring the next spilled partition into
memory while still holding onto memory from the current partition.
The fix is to return earlier if the output batch is at capacity so
that resources are flushed.

Also reduce some of the redundancy in the loop that drives the spilling
logic.

Testing:
The failure was originally reproduced by my IMPALA-4703 patch. I was
able to cause a query failure with the current code by reducing the
memory limit for an existing query. Before it failed with up to 12MB of
memory. Now it succeeds with 8MB or less.

Ran exhaustive build.

Change-Id: I075388d348499c5692d044ac1bc38dd8dd0b10c7
---
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M testdata/workloads/functional-query/queries/QueryTest/spilling.test
3 files changed, 15 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/7180/2
-- 
To view, visit http://gerrit.cloudera.org:8080/7180
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I075388d348499c5692d044ac1bc38dd8dd0b10c7
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>