Return-Path: Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: (qmail 25844 invoked from network); 19 May 2010 20:38:16 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 19 May 2010 20:38:16 -0000 Received: (qmail 28181 invoked by uid 500); 19 May 2010 20:38:16 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 28158 invoked by uid 500); 19 May 2010 20:38:16 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 28150 invoked by uid 99); 19 May 2010 20:38:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 May 2010 20:38:16 +0000 X-ASF-Spam-Status: No, hits=-1445.6 required=10.0 tests=ALL_TRUSTED,AWL X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 May 2010 20:38:14 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o4JKbsWc018330 for ; Wed, 19 May 2010 20:37:54 GMT Message-ID: <5412158.23301274301474269.JavaMail.jira@thor> Date: Wed, 19 May 2010 16:37:54 -0400 (EDT) From: "Ted Yu (JIRA)" To: common-issues@hadoop.apache.org Subject: [jira] Updated: (HADOOP-6774) Namenode is not able to recover from disk full condition In-Reply-To: <27483000.20371274295235326.JavaMail.jira@thor> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HADOOP-6774: --------------------------- Attachment: hadoop-6774.stack Here is the stack trace for job tracker which didn't respond to shutdown request > Namenode is not able to recover from disk full condition > -------------------------------------------------------- > > Key: HADOOP-6774 > URL: https://issues.apache.org/jira/browse/HADOOP-6774 > Project: Hadoop Common > Issue Type: Bug > Components: fs > Affects Versions: 0.20.2 > Environment: Linux sjc9-flash-grid00.ciq.com 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux > Reporter: Ted Yu > Attachments: hadoop-6774.stack > > > We ran an internal flow which resulted in: > Exception in thread "main" java.lang.RuntimeException: initialization of flow executor failed > After that we freed disk space on the Namenode server, but restarting Namenode failed. > Here is from Namenode log: > 2010-05-19 17:15:15,514 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: sjc1-qa-certiq1.sjc1.ciq.com/10.201.8.247:9000 > 2010-05-19 17:15:15,516 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null > 2010-05-19 17:15:15,518 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext > 2010-05-19 17:15:15,579 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop,hadoop > 2010-05-19 17:15:15,579 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup > 2010-05-19 17:15:15,579 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true > 2010-05-19 17:15:15,588 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext > 2010-05-19 17:15:15,590 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean > 2010-05-19 17:15:15,637 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1874 > 2010-05-19 17:15:16,202 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 2 > 2010-05-19 17:15:16,204 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 259450 loaded in 0 seconds. > 2010-05-19 17:15:16,599 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) > at java.lang.Long.parseLong(Long.java:431) > at java.lang.Long.parseLong(Long.java:468) > at org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java:1273) > at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:656) > at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:999) > at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:812) > at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) > at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:312) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:293) > at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:224) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:306) > at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1004) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1013) > 2010-05-19 17:15:16,599 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.