Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201611020527.uA25RToq028559@ip-10-146-233-104.ec2.internal>
Date: Wed, 2 Nov 2016 05:27:29 +0000
From: "Henry Robinson (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org
CC: Marcel Kornacker <marcel@cloudera.com>
Reply-To: henry@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-4415=3A_Fix_unassigned_scan_range_of_size_1=0A?=
In-Reply-To: <gerrit.1478043071000.Id3af767ee9d121ca62ac383ef9e696a18dc903d6@gerrit.cloudera.org>
References: <gerrit.1478043071000.Id3af767ee9d121ca62ac383ef9e696a18dc903d6@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.2
archived-at: Wed, 02 Nov 2016 05:27:42 -0000

Henry Robinson has submitted this change and it was merged.

Change subject: IMPALA-4415: Fix unassigned scan range of size 1
......................................................................


IMPALA-4415: Fix unassigned scan range of size 1

ComputeScanRangeAssignment() computes the per-node scan-range load as
the total number of bytes to be scanned / number of nodes, and casts
that to a float.

However, that computation suffers from a precision issue where the
average * num_nodes may be 1 byte less than the total.

The scan range assignment loop continues until at least average *
num_nodes bytes have been assigned. If the last scan range has only 1
byte, it will not be assigned (if it has 2 or more bytes, it will be
considered for assignment before the loop exit condition is met).

The fix is to make sure that all instances are assigned in the last
iteration of the assignment loop, even if the per-node threshold is
already met.

Testing: No local repro was found - only S3 and LocalFS builds have
it. A unit test requires a lot of infrastructure from
simple-scheduler-util that doesn't exist
yet (e.g. ComputeScanRangeAssignment() is not called). S3 and LocalFS
full test builds are in progress.

Change-Id: Id3af767ee9d121ca62ac383ef9e696a18dc903d6
Reviewed-on: http://gerrit.cloudera.org:8080/4907
Tested-by: Internal Jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
---
M be/src/scheduling/simple-scheduler.cc
1 file changed, 14 insertions(+), 7 deletions(-)

Approvals:
  Henry Robinson: Looks good to me, approved
  Internal Jenkins: Verified


-- 
To view, visit http://gerrit.cloudera.org:8080/4907
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id3af767ee9d121ca62ac383ef9e696a18dc903d6
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Marcel Kornacker <marcel@cloudera.com>