Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 96BA7CF78 for ; Sat, 15 Nov 2014 00:18:37 +0000 (UTC) Received: (qmail 62635 invoked by uid 500); 15 Nov 2014 00:18:35 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 62593 invoked by uid 500); 15 Nov 2014 00:18:35 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 62547 invoked by uid 99); 15 Nov 2014 00:18:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Nov 2014 00:18:35 +0000 Date: Sat, 15 Nov 2014 00:18:35 +0000 (UTC) From: "zhihai xu (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-2816) NM fail to start with NPE during container recovery MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14213133#comment-14213133 ] zhihai xu commented on YARN-2816: --------------------------------- thanks [~jlowe] for the review and commit. > NM fail to start with NPE during container recovery > --------------------------------------------------- > > Key: YARN-2816 > URL: https://issues.apache.org/jira/browse/YARN-2816 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Affects Versions: 2.5.0 > Reporter: zhihai xu > Assignee: zhihai xu > Fix For: 2.7.0 > > Attachments: YARN-2816.000.patch, YARN-2816.001.patch, YARN-2816.002.patch, leveldb_records.txt > > > NM fail to start with NPE during container recovery. > We saw the following crash happen: > 2014-10-30 22:22:37,211 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED; cause: java.lang.NullPointerException > java.lang.NullPointerException > at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverContainer(ContainerManagerImpl.java:289) > at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:252) > at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:235) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:250) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:445) > at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:492) > The reason is some DB files used in NMLeveldbStateStoreService are accidentally deleted to save disk space at /tmp/hadoop-yarn/yarn-nm-recovery/yarn-nm-state. This leaves some incomplete container record which don't have CONTAINER_REQUEST_KEY_SUFFIX(startRequest) entry in the DB. When container is recovered at ContainerManagerImpl#recoverContainer, > The NullPointerException at the following code cause NM shutdown. > {code} > StartContainerRequest req = rcs.getStartRequest(); > ContainerLaunchContext launchContext = req.getContainerLaunchContext(); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)