hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8799) Erasure Coding: add tests for namenode processing corrupt striped blocks
Date Thu, 20 Aug 2015 23:18:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705955#comment-14705955
] 

Zhe Zhang commented on HDFS-8799:
---------------------------------

[~tasanuma0829] Thanks for raising the questions.

bq. And I did some test with your patch. If we want to recover all ec blocks, we need at least
nine datanodes which don't have the corrupt blocks, right?
I think we need 6 DNs hosting healthy blocks (non-corrupt)

bq. So if there are 3 corrupt ec blocks in one ec block group, we need at least 12 datanodes
to recover the all ec blocks. This is not unintuitive. How about add this case?
The test already covers case with 2 corrupt blocks. So I'm fine either way. [~walter.k.su]
The patch needs a minor rebase anyway, please see if you want to add this case.

I see the following issues:
# {{testReportCorruptReplica}} fails
# {{Thread.sleep(3000);}} can be replaced by something similar to {{DFSTestUtil#waitReplication}}
# Similarly, the above could be flaky / slow with a fixed sleep time:
{code}
 // wait 3 seconds for recovery work to finish
 BlockManagerTestUtil.getComputedDatanodeWork(bm);
 Thread.sleep(3000);
{code}
# It seems all tests share the following logic. Maybe we can create a method for that.
{code}
    int length = cellSize * dataBlocks;
    final Path file = new Path("/corrupted");
    createFile(file, length);
    LocatedStripedBlock lsb = (LocatedStripedBlock) fs.getClient()
        .getLocatedBlocks(file.toString(), 0, length).get(0);
{code}
# [follow-on] We should consider consolidating {{corruptBlock}} with the one in {{TestProcessCorruptBlocks}}
and moving it to an util class

> Erasure Coding: add tests for namenode processing corrupt striped blocks
> ------------------------------------------------------------------------
>
>                 Key: HDFS-8799
>                 URL: https://issues.apache.org/jira/browse/HDFS-8799
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: test
>            Reporter: Walter Su
>            Assignee: Walter Su
>            Priority: Minor
>         Attachments: HDFS-8799-HDFS-7285.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message