Return-Path: X-Original-To: apmail-hadoop-yarn-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6DD2910CD2 for ; Thu, 10 Oct 2013 23:52:43 +0000 (UTC) Received: (qmail 97096 invoked by uid 500); 10 Oct 2013 23:52:42 -0000 Delivered-To: apmail-hadoop-yarn-dev-archive@hadoop.apache.org Received: (qmail 97035 invoked by uid 500); 10 Oct 2013 23:52:42 -0000 Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-dev@hadoop.apache.org Delivered-To: mailing list yarn-dev@hadoop.apache.org Received: (qmail 96990 invoked by uid 99); 10 Oct 2013 23:52:42 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Oct 2013 23:52:42 +0000 Date: Thu, 10 Oct 2013 23:52:42 +0000 (UTC) From: "Jian He (JIRA)" To: yarn-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (YARN-1058) Recovery issues on RM Restart with FileSystemRMStateStore MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He resolved YARN-1058. --------------------------- Resolution: Fixed > Recovery issues on RM Restart with FileSystemRMStateStore > --------------------------------------------------------- > > Key: YARN-1058 > URL: https://issues.apache.org/jira/browse/YARN-1058 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager > Affects Versions: 2.1.0-beta > Reporter: Karthik Kambatla > Assignee: Karthik Kambatla > > App recovery doesn't work as expected using FileSystemRMStateStore. > Steps to reproduce: > - Ran sleep job with a single map and sleep time of 2 mins > - Restarted RM while the map task is still running > - The first attempt fails with the following error > {noformat} > Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Password not found for ApplicationAttempt appattempt_1376294441253_0001_000001 > at org.apache.hadoop.ipc.Client.call(Client.java:1404) > at org.apache.hadoop.ipc.Client.call(Client.java:1357) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at $Proxy28.finishApplicationMaster(Unknown Source) > at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:91) > {noformat} > - The second attempt fails with a different error: > {noformat} > Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /tmp/hadoop-yarn/staging/kasha/.staging/job_1376294441253_0001/job_1376294441253_0001_2.jhist: File does not exist. Holder DFSClient_NONMAPREDUCE_389533538_1 does not have any open files. > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2737) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2543) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2454) > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:534) > at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387) > at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:48073) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605) > {noformat} -- This message was sent by Atlassian JIRA (v6.1#6144)