Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 1 Nov 2016 23:31:11 +0000
From: "Henry Robinson (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org
Message-ID: <gerrit.1478043071000.Id3af767ee9d121ca62ac383ef9e696a18dc903d6@gerrit.cloudera.org>
Reply-To: henry@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-4415=3A_Fix_unassigned_scan_range_of_size_1=0A?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.2
archived-at: Tue, 01 Nov 2016 23:31:16 -0000

Henry Robinson has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/4907

Change subject: IMPALA-4415: Fix unassigned scan range of size 1
......................................................................

IMPALA-4415: Fix unassigned scan range of size 1

ComputeScanRangeAssignment() computes the per-node scan-range load as
the total number of bytes to be scanned / number of nodes, and casts
that to a float.

However, that computation suffers from a precision issue where the
average * num_nodes may be 1 byte less than the total.

The scan range assignment loop continues until at least average *
num_nodes bytes have been assigned. If the last scan range has only 1
byte, it will not be assigned (if it has 2 or more bytes, it will be
considered for assignment before the loop exit condition is met).

The fix is to make sure that all instances are assigned in the last
iteration of the assignment loop, even if the per-node threshold is
already met.

Testing: No local repro was found - only S3 and LocalFS builds have
it. A unit test requires a lot of infrastructure from
simple-scheduler-util that doesn't exist
yet (e.g. ComputeScanRangeAssignment() is not called). S3 and LocalFS
full test builds are in progress.

Change-Id: Id3af767ee9d121ca62ac383ef9e696a18dc903d6
---
M be/src/scheduling/simple-scheduler.cc
1 file changed, 14 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/4907/1
-- 
To view, visit http://gerrit.cloudera.org:8080/4907
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id3af767ee9d121ca62ac383ef9e696a18dc903d6
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <henry@cloudera.com>