Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201707260117.v6Q1HEul031543@ip-10-146-233-104.ec2.internal>
Date: Wed, 26 Jul 2017 01:17:14 +0000
From: "Tim Armstrong (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org
CC: Dan Hecht <dhecht@cloudera.com>
Reply-To: tarmstrong@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-4674=3A_Part_3=3A_fix_null-aware_anti_join=0A?=
In-Reply-To: <gerrit.1499379419000.Ie2e60eb4dd32bd287a31479a6232400df65964c1@gerrit.cloudera.org>
References: <gerrit.1499379419000.Ie2e60eb4dd32bd287a31479a6232400df65964c1@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.7
archived-at: Wed, 26 Jul 2017 01:17:20 -0000

Tim Armstrong has uploaded a new patch set (#5).

Change subject: IMPALA-4674: Part 3: fix null-aware anti join
......................................................................

IMPALA-4674: Part 3: fix null-aware anti join

Part 2 regressed NAAJ by tightening up the spilling invariants (e.g.
can't unpin with spilling disabled) but we didn't have tests for
spilling NAAJs that could detect the regression. This patch adds
those tests, fixes the regressions, and improves NAAJ by reliably
spilling the probe side and not trying to bring the whole probe side
into memory.

The changes are:
* All null-aware streams start off in memory and are only unpinned if
  spilling is enabled.
* The null-aware build partition can be spilled in the same way as hash
  partitions.
* Probe streams are unpinned whenever there is memory pressure - if
  spilling is enabled and either a build partition is spilled or
  appending to the probe stream fails.
* Spilled probe streams are not re-pinned in EvaluateNullProbe().
  Instead we just iterate over the rows of the stream.

Testing:
Add query tests where the three different buckets of rows are large
enough to spill: the build and probe of the null-aware partition and the
null probe rows.

Test both spilling and in-memory (with spilling disabled) cases.

Change-Id: Ie2e60eb4dd32bd287a31479a6232400df65964c1
---
M be/src/exec/partitioned-hash-join-builder.cc
M be/src/exec/partitioned-hash-join-builder.h
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
A testdata/workloads/functional-query/queries/QueryTest/spilling-naaj.test
M tests/query_test/test_spilling.py
7 files changed, 422 insertions(+), 91 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/7367/5
-- 
To view, visit http://gerrit.cloudera.org:8080/7367
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie2e60eb4dd32bd287a31479a6232400df65964c1
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>