Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Help - can't start namenode after disk full error
Date: Sun, 12 Jun 2011 10:47:29 -0400
Message-ID: 
 <34E9B7D077A04640BDFB43DF226509C20432D573@ustrysvpexch23.kih.kmart.com>
In-Reply-To: 
 <34E9B7D077A04640BDFB43DF226509C20432D572@ustrysvpexch23.kih.kmart.com>
Thread-Topic: Help - can't start namenode after disk full error
Thread-Index: AcwpDTKkIHh8m14URQyDxldVWSOBeAAAWKxwAAA6ISA=
References: <BANLkTi=d1=1Rmyp9qsXB_NWtMn1MwN=2Cw@mail.gmail.com>
 <34E9B7D077A04640BDFB43DF226509C20432D572@ustrysvpexch23.kih.kmart.com>
From: "Zhong, Andy" <Sheng.Zhong@searshc.com>
To: <common-user@hadoop.apache.org>

Only apply it to /dfs/name/current/edits file...

-----Original Message-----
From: Zhong, Andy [mailto:Sheng.Zhong@searshc.com]=20
Sent: Sunday, June 12, 2011 9:43 AM
To: common-user@hadoop.apache.org
Subject: RE: Help - can't start namenode after disk full error

Could you backup your edits file, try $ printf "\xff\xff\xff\xee\xff" >
edits, and start HDFS? It should work. - Andy=20

-----Original Message-----
From: Ryan LeCompte [mailto:lecompte@gmail.com]
Sent: Sunday, June 12, 2011 9:29 AM
To: common-user@hadoop.apache.org
Subject: Help - can't start namenode after disk full error

Hey guys,

Really trying to get our namenode back up and running after a full disk
error last night. I've freed up a lot of space, however the NameNode
still fails to startup:

2011-06-12 10:26:09,042 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2011-06-12 10:26:09,083 INFO
org.apache.hadoop.hdfs.server.common.Storage:
Number of files =3D 614919
2011-06-12 10:26:22,293 INFO
org.apache.hadoop.hdfs.server.common.Storage:
Number of files under construction =3D 17
2011-06-12 10:26:22,300 INFO
org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 102029859 loaded in 13 seconds.
2011-06-12 10:26:22,510 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NumberFormatException: For input string: ""
at
java.lang.NumberFormatException.forInputString(NumberFormatException.jav
a:48)
at java.lang.Integer.parseInt(Integer.java:468)
at java.lang.Short.parseShort(Short.java:120)
at java.lang.Short.parseShort(Short.java:78)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readShort(FSEditLog.jav
a:1269)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
ava:550)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
992)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
812)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
mage.java:364)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
ry.java:87)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
tem.java:311)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
java:292)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
:201)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)


We currently have our config setup as follows:

 <property>
    <name>dfs.name.dir</name>

<value>/data1/hadoop/dfs/name,/data2/hadoop/dfs/name,/data3/hadoop/dfs/n
ame,/data4/hadoop/dfs/name</value>
  </property>

I've looked in each of those directories for an image/edits.new file,
but only the "edits" files exist.

Can anyone please guide me on the next step here to get this back up and
running?

Thanks!

Ryan

This message, including any attachments, is the property of Sears
Holdings Corporation and/or one of its subsidiaries. It is confidential
and may contain proprietary or legally privileged information. If you
are not the intended recipient, please delete it without reading the
contents. Thank you.