hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Namenode EOF Exception
Date Mon, 14 May 2012 18:16:13 GMT
Your fsimage seems to have gone bad (is it 0-sized? I recall that as a
known issue long since fixed).

The easiest way is to fall back to the last available good checkpoint
(From SNN). Or if you have multiple dfs.name.dirs, see if some of the
other points have better/complete files on them, and re-spread them
across after testing them out (and backing up the originals).

Though what version are you running? Cause AFAIK most of the recent
stable versions/distros include NN resource monitoring threads which
should have placed your NN into safemode the moment all its disks ran
near to out of space.

On Mon, May 14, 2012 at 10:50 PM, Prashant Kommireddi
<prash1784@gmail.com> wrote:
> Hi,
>
> I am seeing an issue where Namenode does not start due an EOFException. The
> disk was full and I cleared space up but I am unable to get past this
> exception. Any ideas on how this can be resolved?
>
> 2012-05-14 10:10:44,018 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=hadoop
> 2012-05-14 10:10:44,018 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=false
> 2012-05-14 10:10:44,023 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
> Initializing FSNamesystemMetrics using context
> object:org.apache.hadoop.metrics.file.FileContext
> 2012-05-14 10:10:44,024 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> 2012-05-14 10:10:44,047 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Number of files = 205470
> 2012-05-14 10:10:44,844 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
> initialization failed.
> java.io.EOFException
>    at java.io.DataInputStream.readFully(DataInputStream.java:180)
>    at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSImage.readString(FSImage.java:1578)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:880)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:807)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)
>    at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
>    at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
>    at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>    at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
> 2012-05-14 10:10:44,845 INFO org.apache.hadoop.ipc.Server: Stopping server
> on 54310
> 2012-05-14 10:10:44,845 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.EOFException
>    at java.io.DataInputStream.readFully(DataInputStream.java:180)
>    at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSImage.readString(FSImage.java:1578)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:880)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:807)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
>    at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)
>    at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
>    at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
>    at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>    at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> 2012-05-14 10:10:44,846 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at
> gridforce-1.internal.salesforce.com/10.0.201.159
> ************************************************************/



-- 
Harsh J

Mime
View raw message