Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Wed, 8 Oct 2014 21:01:34 +0000 (UTC)
From: "zhihai xu (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12745665.1412300047000.221673.1412802094369@Atlassian.JIRA>
In-Reply-To: <JIRA.12745665.1412300047000@Atlassian.JIRA>
References: <JIRA.12745665.1412300047000@Atlassian.JIRA>
 <JIRA.12745665.1412300047351@arcas>
Subject: [jira] [Commented] (YARN-2641) improve node decommission latency in
 RM.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164152#comment-14164152 ] 

zhihai xu commented on YARN-2641:
---------------------------------

Hi [~djp], thanks for the explanation, this is a good discussion.

The patch can address the other scenario:
when RM receive decommission CLI, the RM can't receive the NM's heartbeat to send shutdown command to the NM, which is to be  decommissioned, because the NM to be decommissioned is killed right after RM receive decommission CLI. In this scenario, The RMNode will never be decommissioned in RM. The RMNode will only expire(RMNodeEventType.EXPIRE) in RM(NMLivelinessMonitor) after "yarn.nm.liveness-monitor.expiry-interval-ms"(default value 10 minutes) time.

About the first scenario:
for attempt scheduling case, before create this JIRA, I think about to check whether the node is valid in the scheduler before allocate container at the node. But I think this may not be a good way because the scheduler is always the bottleneck in RM, if we can offload the work from scheduler, it would be better to do that.

what do you think? thanks

> improve node decommission latency in RM.
> ----------------------------------------
>
>                 Key: YARN-2641
>                 URL: https://issues.apache.org/jira/browse/YARN-2641
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.5.0
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>         Attachments: YARN-2641.000.patch, YARN-2641.001.patch
>
>
> improve node decommission latency in RM. 
> Currently the node decommission only happened after RM received nodeHeartbeat from the Node Manager. The node heartbeat interval is configurable. The default value is 1 second.
> It will be better to do the decommission during RM Refresh(NodesListManager) instead of nodeHeartbeat(ResourceTrackerService).
> This will be a much more serious issue:
> After RM is refreshed (refreshNodes), If the NM to be decommissioned is killed before NM sent heartbeat to RM. The RMNode will never be decommissioned in RM. The RMNode will only expire in RM after  "yarn.nm.liveness-monitor.expiry-interval-ms"(default value 10 minutes) time.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)