hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3384) DataStreamer thread should be closed immediatly when failed to setup a PipelineForAppendOrRecovery
Date Mon, 28 May 2012 05:49:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284295#comment-13284295
] 

Uma Maheswara Rao G commented on HDFS-3384:
-------------------------------------------

Hi Amith, Change looks good to me.

nits: 
{code}
DistributedFileSystem fs = cluster.getFileSystem();
+    Path fileName = new Path("/appendCorruptBlock");
+    DFSTestUtil.createFile(fs, fileName, 512, (short) 1, 0);
+    DFSTestUtil.waitReplication(fs, fileName, (short)1);
+    Assert.assertTrue("File not created", fs.exists(fileName));
+    /*
+     * Corrupt the blocks in the cluster
+     */
+    corruptBlockInFinalizedDirectory(cluster);{code}
Here null check for cluster as you have instantiated out side of try. And Better move other
code into try after creating cluster. check the other places as well in corruptFinalizedBlock.

2) "Finalized directory not created yet"
used in multiple places. create constant and use.

3) con't we move this test to some existing append test files instead of creating your own
test file for single test case?

Nicholas, do you have any comments? 
I agree that, other than adding assertion in code, I could not find other good way to test
this, as Streamer is handling Throwable internally. I am not sure, you will agree with this,
please give your opinion. 

                
> DataStreamer thread should be closed immediatly when failed to setup a PipelineForAppendOrRecovery
> --------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3384
>                 URL: https://issues.apache.org/jira/browse/HDFS-3384
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 2.0.0-alpha
>            Reporter: Brahma Reddy Battula
>            Assignee: amith
>         Attachments: HDFS-3384.patch, HDFS-3384_2.patch, HDFS-3384_2.patch
>
>
> Scenraio:
> =========
> write a file
> corrupt block manually
> call append..
> {noformat}
> 2012-04-19 09:33:10,776 INFO  hdfs.DFSClient (DFSOutputStream.java:createBlockOutputStream(1059))
- Exception in createBlockOutputStream
> java.io.EOFException: Premature EOF: no length prefix available
> 	at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1039)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:939)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> 2012-04-19 09:33:10,807 WARN  hdfs.DFSClient (DFSOutputStream.java:run(549)) - DataStreamer
Exception
> java.lang.NullPointerException
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:510)
> 2012-04-19 09:33:10,807 WARN  hdfs.DFSClient (DFSOutputStream.java:hflush(1511)) - Error
while syncing
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message