hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Prakash (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2848) hdfs corruption appended to blocks is not detected by fs commands or fsck
Date Thu, 02 Feb 2012 22:22:54 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199285#comment-13199285
] 

Ravi Prakash commented on HDFS-2848:
------------------------------------

Ran append on a corrupted file. The corruption is simply overwritten. So we're good on that
front. If the corrupted data is longer than the appended data, then the corrupt data still
extends beyond the correct data.

If I restart the DN, and then try appending again, it detects the corruption successfully.
{noformat}
$ java TestHadoop23 #My test append program
2012-02-02 16:18:44,083 INFO  hdfs.DFSClient (DFSOutputStream.java:createBlockOutputStream(1059))
- Exception in createBlockOutputStream
java.io.EOFException: Premature EOF: no length prefix available
        at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:158)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1038)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:939)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
2012-02-02 16:18:44,087 WARN  hdfs.DFSClient (DFSOutputStream.java:run(549)) - DataStreamer
Exception
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:510)
Exception in thread "main" java.io.IOException: All datanodes <someip>:50010 are bad.
Aborting...
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:907)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
{noformat}
                
> hdfs corruption appended to blocks is not detected by fs commands or fsck
> -------------------------------------------------------------------------
>
>                 Key: HDFS-2848
>                 URL: https://issues.apache.org/jira/browse/HDFS-2848
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Ravi Prakash
>            Assignee: Ravi Prakash
>
> Courtesy Pat White [~patwhitey2007]
> {quote}
> Appears that there is a regression in corrupt block detection by both fsck and fs cmds
like 'cat'. Testcases for
> pre-block and block-overwrite corruption of all replicas is correctly reporting errors
however post-block corruption is
> not, fsck on the filesystem reports it's Healthy and 'cat' returns without error. Looking
at the DN blocks themselves,
> they clearly contain the injected corruption pattern.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message