impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henry Robinson (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4415: Fix unassigned scan range of size 1
Date Wed, 02 Nov 2016 01:02:27 GMT
Hello Marcel Kornacker,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/4907

to look at the new patch set (#2).

Change subject: IMPALA-4415: Fix unassigned scan range of size 1
......................................................................

IMPALA-4415: Fix unassigned scan range of size 1

ComputeScanRangeAssignment() computes the per-node scan-range load as
the total number of bytes to be scanned / number of nodes, and casts
that to a float.

However, that computation suffers from a precision issue where the
average * num_nodes may be 1 byte less than the total.

The scan range assignment loop continues until at least average *
num_nodes bytes have been assigned. If the last scan range has only 1
byte, it will not be assigned (if it has 2 or more bytes, it will be
considered for assignment before the loop exit condition is met).

The fix is to make sure that all instances are assigned in the last
iteration of the assignment loop, even if the per-node threshold is
already met.

Testing: No local repro was found - only S3 and LocalFS builds have
it. A unit test requires a lot of infrastructure from
simple-scheduler-util that doesn't exist
yet (e.g. ComputeScanRangeAssignment() is not called). S3 and LocalFS
full test builds are in progress.

Change-Id: Id3af767ee9d121ca62ac383ef9e696a18dc903d6
---
M be/src/scheduling/simple-scheduler.cc
1 file changed, 14 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/4907/2
-- 
To view, visit http://gerrit.cloudera.org:8080/4907
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id3af767ee9d121ca62ac383ef9e696a18dc903d6
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <marcel@cloudera.com>

Mime
View raw message