hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6420) RM startup failure due to wrong order in nodelabel editlog
Date Mon, 03 Apr 2017 10:48:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953267#comment-15953267

Varun Saxena commented on YARN-6420:

Thanks [~bibinchundatt] for the patch.
You have added write locks in multiple methods of CommonNodeLabelsManager. But adding a write
lock in only CommonNodeLabelsManager#addToCluserNodeLabels would be sufficient to fix this
issue as for other node label operations come via RMNodeLabelManager which acquires the same
write lock already.

However, CommonNodeLabelsManager class if taken in isolation and the fact that its not an
abstract class means that we can actually make the locking consistent there as well.
So I am fine with what you have done in the patch.

Let me look at the patch further in detail later today.

> RM startup failure due to wrong order in nodelabel editlog
> ----------------------------------------------------------
>                 Key: YARN-6420
>                 URL: https://issues.apache.org/jira/browse/YARN-6420
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Bibin A Chundatt
>            Assignee: Bibin A Chundatt
>            Priority: Critical
>         Attachments: YARN-6420.0001.patch
> Edit log file for nodelabel written in wrong order if {{StoreNewClusterNodeLabels}} addition
is delayed and {{UpdateNodeToLabelsMappingsEvent}} is added to dispatcher.
> Configure RM admin client thread count to 2
> Add node label to cluster X - Client 1
> Delay event addition to dispatcher 
> Replace node label on node1  to X - Client 2
> Make sure {{UpdateNodeToLabelsMappingsEvent}} added to dispatcher first.
> Restart  resource manager
> {noformat}
> 2017-03-31 16:20:42,236 | WARN  | main-EventThread | Exception handling the winning of
election | ActiveStandbyElector.java:836
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
>         at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:128)
>         at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:832)
>         at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:422)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:728)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:600)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error when transitioning to Active
>         at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:331)
>         at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
>         ... 4 more
> Caused by: org.apache.hadoop.service.ServiceStateException: java.io.IOException: Not
all labels being replaced contained by known label collections, please check, new labels=[1]
>         at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
>         at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
>         at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
>         at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:642)
>         at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>         at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1042)
>         at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1083)
>         at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1079)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
> {noformat}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message