hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sravankorumilli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1887) Facing problems while restarting the datanode if the datanode format is unsuccessful.
Date Thu, 26 May 2011 07:33:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039558#comment-13039558

sravankorumilli commented on HDFS-1887:

I want to propose this solution:-

In order to avoid manual intervention of deleting the storage directory we can check for a
file say "formatrequired" file if the file is present we will format the storage directory
The file will be created while formatting a storage directory is started and deleted once
the version file is written successfully. This will ensure formatting has been done successfully.
Whenever datanode is started the first step of the analyzing the storage directory is to check
the "formatrequired" file is present or not if the file is present then we will set the start
up option to format then the storage directory will be formatted and the datanode will be
started successfully.

> Facing problems while restarting the datanode if the datanode format is unsuccessful.
> -------------------------------------------------------------------------------------
>                 Key: HDFS-1887
>                 URL: https://issues.apache.org/jira/browse/HDFS-1887
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.20.1, 0.21.0, 0.23.0
>         Environment: Linux
>            Reporter: sravankorumilli
> In the existing behavior we are checking whether datanode is formatted or not based on
the version file existence. If version file is not present the storage directory will be formatted.
In some cases if formatting got terminated abruptly there can be a scenario where storage
file or version file will be created and the content may not be written. In such scenarios
when Datanode is restarted it is just throwing an exception. Some one has to manually delete
the storage directory and restart the datanode.
> This is one scenario where storage file is created but the content is not written then
I am getting this exception.
> These are the corresponding logs:-
> 2011-05-02 19:12:19,389 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at org.apache.hadoop.hdfs.server.datanode.DataStorage.isConversionNeeded(DataStorage.java:203)
> at org.apache.hadoop.hdfs.server.common.Storage.checkConversionNeeded(Storage.java:697)
> at org.apache.hadoop.hdfs.server.common.Storage.access$000(Storage.java:62)
> at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:476)
> at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:116)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:336)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:260)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:237)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1440)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1393)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1407)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1552)
> Our Hadoop cluster is managed by a cluster management software which tries to eliminate
any manual intervention in setting up & managing the cluster. But in the above mentioned
scenario, it requires manual intervention to recover the DataNode.Though it is very rare there
is a possibility for this

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message