hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi Ozawa (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-4348) ZKRMStateStore.syncInternal should wait for zkResyncWaitTime instead of zkSessionTimeout
Date Thu, 12 Nov 2015 00:24:11 GMT
Tsuyoshi Ozawa created YARN-4348:
------------------------------------

             Summary: ZKRMStateStore.syncInternal should wait for zkResyncWaitTime instead
of zkSessionTimeout
                 Key: YARN-4348
                 URL: https://issues.apache.org/jira/browse/YARN-4348
             Project: Hadoop YARN
          Issue Type: Bug
    Affects Versions: 2.7.2
            Reporter: Tsuyoshi Ozawa
            Assignee: Tsuyoshi Ozawa


The current internal ZK configuration of ZKRMStateStore can cause a following situation:

1. syncInternal timeouts, 
2. but sync succeeded later on.

{quote}
2015-11-11 11:54:05,728 INFO  recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1241))
- Failed to sync with ZK new connection.  -<--- sync failed
2015-11-11 11:54:05,728 INFO  recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1244))
- Maxed out ZK retries. Giving up!
2015-11-11 11:54:05,728 ERROR recovery.RMStateStore (RMStateStore.java:transition(292)) -
Error updating appAttempt: appattempt_1447242474882_0002_000001
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
for /rmstore/ZKRMStateRoot/RMAppRoot/application_1447242474882_0002/appattempt_1447242474882_0002_000001
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
  at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$9.run(ZKRMStateStore.java:1082)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$9.run(ZKRMStateStore.java:1079)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1164)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1197)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.existsWithRetries(ZKRMStateStore.java:1079)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationAttemptStateInternal(ZKRMStateStore.java:716)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:286)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:269)
  at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
  at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
  at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
  at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:1006)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1075)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1070)
  at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
  at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
  at java.lang.Thread.run(Thread.java:745)
2015-11-11 11:54:05,729 ERROR recovery.RMStateStore (RMStateStore.java:notifyStoreOperationFailed(1027))
- State store operation failed
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
for /rmstore/ZKRMStateRoot/RMAppRoot/application_1447242474882_0002/appattempt_1447242474882_0002_000001
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
  at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$9.run(ZKRMStateStore.java:1082)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$9.run(ZKRMStateStore.java:1079)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1164)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1197)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.existsWithRetries(ZKRMStateStore.java:1079)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationAttemptStateInternal(ZKRMStateStore.java:716)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:286)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:269)
  at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
  at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
  at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
  at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:1006)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1075)
  at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1070)
  at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
  at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
  at java.lang.Thread.run(Thread.java:745)
2015-11-11 11:54:05,730 WARN  recovery.RMStateStore (RMStateStore.java:notifyStoreOperationFailed(1041))
- Skip the state-store error.
2015-11-11 11:54:05,730 INFO  recovery.RMStateStore (RMStateStore.java:transition(206)) -
Updating info for app: application_1447242474882_0002
2015-11-11 11:54:06,018 INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(726)) - application_1447242474882_0002
State change from FINAL_SAVING to FINISHING
2015-11-11 11:54:06,018 INFO  recovery.ZKRMStateStore (ZKRMStateStore.java:processWatchEvent(913))
- Watcher event type: None with state:SyncConnected for path:null for Service org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore
in state org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: STARTED
2015-11-11 11:54:06,019 INFO  recovery.ZKRMStateStore (ZKRMStateStore.java:processWatchEvent(921))
- ZKRMStateStore Session connected
2015-11-11 11:54:06,019 INFO  recovery.ZKRMStateStore (ZKRMStateStore.java:processResult(123))
- ZooKeeper sync operation succeeded. path: /rmstore/ZKRMStateRoot/RMAppRoot/application_1447242474882_0002/appattempt_1447242474882_0002_000001
 <- here, sync succeeded.
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message