hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhihai xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2641) improve node decommission latency in RM.
Date Sun, 12 Oct 2014 08:11:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168564#comment-14168564
] 

zhihai xu commented on YARN-2641:
---------------------------------

[~kasha] thanks to review the patch.
Just synchronized hostsReader in NodeListManager#isValidNode is not enough.
There is a race condition between ResourceTrackerService#registerNodeManager and NodeListManager#refreshNodes:
If NodeListManager#refreshNodes happens right after NodeListManager#isValidNode and before
create a new RMNode in ResourceTrackerService#registerNodeManager.
The node to be decommissioned will be added after NodeListManager#refreshNodes.
And it will never be decommissioned until next time NodeListManager#refreshNodes is called.

By synchronizing isValidNode and create-a-new-RMNode on hostsReader in ResourceTrackerService#registerNodeManager,
we can make sure the NodeListManager#refreshNodes is called either before isValidNode or after
create-a-new-RMNode.
If it is called before check the node, isValidNode will return false which will shutdown the
node.
If it is called after RMNode is created, the new created RMNode will be decommissioned by
NodeListManager#refreshNodes immediately.

> improve node decommission latency in RM.
> ----------------------------------------
>
>                 Key: YARN-2641
>                 URL: https://issues.apache.org/jira/browse/YARN-2641
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.5.0
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>         Attachments: YARN-2641.000.patch, YARN-2641.001.patch
>
>
> improve node decommission latency in RM. 
> Currently the node decommission only happened after RM received nodeHeartbeat from the
Node Manager. The node heartbeat interval is configurable. The default value is 1 second.
> It will be better to do the decommission during RM Refresh(NodesListManager) instead
of nodeHeartbeat(ResourceTrackerService).
> This will be a much more serious issue:
> After RM is refreshed (refreshNodes), If the NM to be decommissioned is killed before
NM sent heartbeat to RM. The RMNode will never be decommissioned in RM. The RMNode will only
expire in RM after  "yarn.nm.liveness-monitor.expiry-interval-ms"(default value 10 minutes)
time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message