hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3644) Node manager shuts down if unable to connect with RM
Date Fri, 10 Jul 2015 15:24:08 GMT

    [ https://issues.apache.org/jira/browse/YARN-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14622459#comment-14622459

Varun Saxena commented on YARN-3644:

Thanks [~raju.bairishetti] for working on this.

FTew comments :
# No need to override {{MyNodeManager3#getNodeStatusUpdater}} and {{MyNodeManager3#serviceStop}}.
serviceStop for instance only calls {{super.serviceStop()}}
# {{MyNodeStatusUpdater6#context}} is not required.
# The test doesnt really check for whether ConnectionException was thrown or NM Shutdown event
was called or not. I think you check whether the flow has been hit or not using Mockito mock
or spy. For instance, if you call {{NodeManager#start}}, service state will be STARTED irrespective
of whether code written by you has been hit or not.
# I think log added can be logged at WARN level instead of ERROR.
# Also the log says "Not shutting down NodeManager. Retry after default heartbeat interval
time". We can instead say something like "Unable to connect to RM...Retry after default heartbeat
# The config name is {{yarn.nodemanager.shutdown.on.RM.connection.failures}}. All our config
names are in lowercase, just for the sake of consistency, maybe RM can be in lowercase too.
Thoughts ? 

> Node manager shuts down if unable to connect with RM
> ----------------------------------------------------
>                 Key: YARN-3644
>                 URL: https://issues.apache.org/jira/browse/YARN-3644
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Srikanth Sundarrajan
>            Assignee: Raju Bairishetti
>         Attachments: YARN-3644.001.patch, YARN-3644.001.patch, YARN-3644.002.patch, YARN-3644.003.patch,
> When NM is unable to connect to RM, NM shuts itself down.
> {code}
>           } catch (ConnectException e) {
>             //catch and throw the exception if tried MAX wait time to connect RM
>             dispatcher.getEventHandler().handle(
>                 new NodeManagerEvent(NodeManagerEventType.SHUTDOWN));
>             throw new YarnRuntimeException(e);
> {code}
> In large clusters, if RM is down for maintenance for longer period, all the NMs shuts
themselves down, requiring additional work to bring up the NMs.
> Setting the yarn.resourcemanager.connect.wait-ms to -1 has other side effects, where
non connection failures are being retried infinitely by all YarnClients (via RMProxy).

This message was sent by Atlassian JIRA

View raw message