Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Mon, 14 Dec 2015 22:57:46 +0000 (UTC)
From: "zhihai xu (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12920776.1449757552000.19822.1450133866712@Atlassian.JIRA>
In-Reply-To: <JIRA.12920776.1449757552000@Atlassian.JIRA>
References: <JIRA.12920776.1449757552000@Atlassian.JIRA>
 <JIRA.12920776.1449757552803@arcas>
Subject: [jira] [Commented] (YARN-4440)
 FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler
 time
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056895#comment-15056895 ] 

zhihai xu commented on YARN-4440:
---------------------------------

Good catch! thanks for working on this issue [~linyiqun]!
+1 for the latest patch, The test failures are not related to the patch, These failures were already reported at YARN-4318 and YARN-4306.
Will commit it tomorrow if no one objects.


> FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time
> -----------------------------------------------------------------------------
>
>                 Key: YARN-4440
>                 URL: https://issues.apache.org/jira/browse/YARN-4440
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.7.1
>            Reporter: Lin Yiqun
>            Assignee: Lin Yiqun
>         Attachments: YARN-4440.001.patch, YARN-4440.002.patch, YARN-4440.003.patch
>
>
> It seems there is a bug on {{FSAppAttempt#getAllowedLocalityLevelByTime}} method
> {code}
> // default level is NODE_LOCAL
>     if (! allowedLocalityLevel.containsKey(priority)) {
>       allowedLocalityLevel.put(priority, NodeType.NODE_LOCAL);
>       return NodeType.NODE_LOCAL;
>     }
> {code}
> If you first invoke this method, it doesn't init  time in lastScheduledContainer and this will lead to execute these code for next invokation:
> {code}
>     // check waiting time
>     long waitTime = currentTimeMs;
>     if (lastScheduledContainer.containsKey(priority)) {
>       waitTime -= lastScheduledContainer.get(priority);
>     } else {
>       waitTime -= getStartTime();
>     }
> {code}
> the waitTime will subtract to FsApp startTime, and this will be easily more than the delay time and allowedLocality degrade. Because FsApp startTime will be start earlier than currentTimeMs. So we should add the initial time of priority to prevent comparing with FsApp startTime and allowedLocalityLevel degrade. And this problem will have more negative influence for small-jobs. The YARN-4399 also discuss some problem in aspect of locality.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)