hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small
Date Wed, 05 Aug 2015 01:35:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654654#comment-14654654

Tsz Wo Nicholas Sze commented on HDFS-8838:

[~walter.k.su], thanks for running the tests.  I tried to use the same MiniDFSCluster to run
all the tests so that I had changed DN dir names.  Otherwise, the dir used by an old datanode
cannot be re-used by a new datanode.  It needs more work since some other tests failed.  
 TestBase not public is fine in my machine.  Not sure why it does not work for yours.  Reverted
both for the moment.

> 3. The numBytes returned from getBlockGroup() is wrong. We should use the old method,
which is to use streamer #0 to check block boundry, but not other streamers, as it used to
be before the patch.

I changed it to check cell boundary instead of block group boundary.  If we use the old method,
some of the new tests fail.  However, TestDFSStripedOutputStream fails after the change. I
will see what's wrong.

> Tolerate datanode failures in DFSStripedOutputStream when the data length is small
> ----------------------------------------------------------------------------------
>                 Key: HDFS-8838
>                 URL: https://issues.apache.org/jira/browse/HDFS-8838
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: h8838_20150729.patch, h8838_20150731-HDFS-7285.patch, h8838_20150731.log,
h8838_20150731.patch, h8838_20150804.patch
> Currently, DFSStripedOutputStream cannot tolerate datanode failures when the data length
is small.  We fix the bugs here and add more tests.

This message was sent by Atlassian JIRA

View raw message