hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7453) Fix issue where RM fails to switch to active after first successful start
Date Thu, 09 Nov 2017 03:27:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245154#comment-16245154
] 

Rohith Sharma K S commented on YARN-7453:
-----------------------------------------

Thanks [~asuresh] for committing the patch. I thank all of you for helping investigating the
issue and verifying the patch. Special thanks to [~jianhe] [~sunilg] for helping isolating
the problem, quick build verifications and for initial patch Sunil provided to me offline!

> Fix issue where RM fails to switch to active after first successful start
> -------------------------------------------------------------------------
>
>                 Key: YARN-7453
>                 URL: https://issues.apache.org/jira/browse/YARN-7453
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.9.0, 3.1.0
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>            Priority: Blocker
>             Fix For: 2.9.0, 3.0.0, 3.1.0
>
>         Attachments: YARN-7453.001.patch, YARN-7453.001.patch
>
>
> It is observed that RM fail to switch to ACTIVE after first successful start! The below
exception throws when RM is switching from ACTIVE->STANDBY->ACTIVE. This continues in
loop!
> {noformat}
> 2017-11-07 15:08:11,664 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager:
Transitioning to active state
> 2017-11-07 15:08:11,669 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager:
Recovery started
> 2017-11-07 15:08:11,669 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
Loaded RM state version info 1.5
> 2017-11-07 15:08:11,670 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager:
Failed to load/recover state
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> 	at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1006)
> 	at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> 	at org.apache.curator.framework.imps.CuratorTransactionImpl.doOperation(CuratorTransactionImpl.java:159)
> 	at org.apache.curator.framework.imps.CuratorTransactionImpl.access$200(CuratorTransactionImpl.java:44)
> 	at org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:129)
> 	at org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:125)
> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> 	at org.apache.curator.framework.imps.CuratorTransactionImpl.commit(CuratorTransactionImpl.java:122)
> 	at org.apache.hadoop.util.curator.ZKCuratorManager$SafeTransaction.commit(ZKCuratorManager.java:403)
> 	at org.apache.hadoop.util.curator.ZKCuratorManager.safeSetData(ZKCuratorManager.java:372)
> 	at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.getAndIncrementEpoch(ZKRMStateStore.java:493)
> 	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:771)
> 	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> 	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1162)
> 	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1202)
> 	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1198)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> 	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1198)
> 	at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:320)
> 	at org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> 	at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:894)
> 	at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:473)
> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:607)
> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message