hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma" <jssa...@facebook.com>
Subject RE: missing VERSION files leading to failed datanodes
Date Tue, 08 Jan 2008 17:34:14 GMT
well - at least i know why this happened. (still looking for a way to restore the version file).

https://issues.apache.org/jira/browse/HADOOP-2549 is causing disk full on one of the disks
(in spite of du.reserved setting). looks like while starting up - the VERSION file could not
be written and went missing. that would seem like another bug (writing a tmp file and renaming
it to VERSION file would have prevented this mishap):

2008-01-08 08:24:01,597 ERROR org.apache.hadoop.dfs.DataNode: java.io.IOException: No space
left on device
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:260)
        at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
        at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
        at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:408)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
        at java.io.BufferedWriter.flush(BufferedWriter.java:236)
        at java.util.Properties.store(Properties.java:666)
        at org.apache.hadoop.dfs.Storage$StorageDirectory.write(Storage.java:176)
        at org.apache.hadoop.dfs.Storage$StorageDirectory.write(Storage.java:164)
        at org.apache.hadoop.dfs.Storage.writeAll(Storage.java:510)
        at org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:146)
        at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:243)

-----Original Message-----
From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
Sent: Tue 1/8/2008 8:51 AM
To: hadoop-user@lucene.apache.org
Subject: missing VERSION files leading to failed datanodes

2008-01-08 08:36:20,045 ERROR org.apache.hadoop.dfs.DataNode: org.apache.hadoop.dfs.InconsistentFSStateException:
Directory /var/hadoop/tmp/dfs/data is in an inconsistent state: file VERSION is invalid.

[root@hadoop034.sf2p data]# ssh hadoop003.sf2p cat /var/hadoop/tmp/dfs/data/current/VERSION

[root@hadoop034.sf2p data]# 

any idea why the VERSION file is empty? and how can i regenerate it - or ask the system to
generate a new one without discarding all the blocks?

i had previously shutdown and started dfs once (to debug a different bug where it's not honoring
du.reserved. more on that later).

help appreciated,


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message