hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
Date Thu, 21 Feb 2013 19:48:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583480#comment-13583480

Hadoop QA commented on YARN-412:

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 2 new or modified
test files.

  {color:red}-1 one of tests included doesn't have a timeout.{color}

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/415//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/415//console

This message is automatically generated.
> FifoScheduler incorrectly checking for node locality
> ----------------------------------------------------
>                 Key: YARN-412
>                 URL: https://issues.apache.org/jira/browse/YARN-412
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: Roger Hoover
>            Priority: Minor
>              Labels: patch
>         Attachments: YARN-412.patch
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the data is
local to a node by searching for the nodeAddress of the node in the set of outstanding requests
for the app.  This seems to be incorrect as it should be checking hostname instead.  The offending
line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses are a concatenation
of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See LeafQueue.assignNodeLocalContainers,
line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by the FifoScheduler
because even though it incorrect determines that a request is not local to the node, it will
still schedule the request immediately because it's rack-local.  However, this bug may be
adversely affecting the reporting of job status by underreporting the number of tasks that
were node local.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message