hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8120) Erasure coding: created util class to analyze striped block groups
Date Tue, 14 Apr 2015 04:29:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14493543#comment-14493543
] 

Jing Zhao commented on HDFS-8120:
---------------------------------

Thanks for updating the patch, Zhe. And thanks for the review, Walter. The 002 patch looks
good to me. The only comment is about the changes in {{BlockManager#checkReplicaCorrupt}}.
{code}
+        assert reported.getBlockId() - storedBlock.getBlockId() <
+            HdfsConstants.MAX_BLOCKS_IN_GROUP;
+        byte reportedBlkIdx = (byte)
+            (reported.getBlockId() - storedBlock.getBlockId());
         if (storedBlock.getGenerationStamp() != reported.getGenerationStamp()) {
           final long reportedGS = reported.getGenerationStamp();
           return new BlockToMarkCorrupt(new Block(reported), storedBlock, reportedGS,
               "block is " + ucState + " and reported genstamp " + reportedGS
               + " does not match genstamp in block map "
               + storedBlock.getGenerationStamp(), Reason.GENSTAMP_MISMATCH);
-        } else if (storedBlock.getNumBytes() != reported.getNumBytes()) {
+        } else if (StripedBlockUtil.getInternalBlockLength(
+            storedBlock.getNumBytes(), HdfsConstants.BLOCK_STRIPED_CELL_SIZE,
+            HdfsConstants.NUM_DATA_BLOCKS, reportedBlkIdx)
+            != reported.getNumBytes()) {
{code}
More specifically,
# The assert should be only for striped blocks. Also {{MAX_BLOCKS_IN_GROUP}} can be replaced
by {{storedBlock.getTotalBlockNum()}}
# Let's use {{BlockIdManager.getBlockIndex}} to get internal block index
# The "else if" part should cover both striped and contiguous blocks
# Instead of using {{HdfsConstants.NUM_DATA_BLOCKS}}, we should use {{storedBlock.getDataBlockNum}}
# We need to have some unit tests for this block report fix

> Erasure coding: created util class to analyze striped block groups
> ------------------------------------------------------------------
>
>                 Key: HDFS-8120
>                 URL: https://issues.apache.org/jira/browse/HDFS-8120
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8120.000.patch, HDFS-8120.001.patch, HDFS-8120.002.patch
>
>
> The patch adds logic of calculating size of individual blocks in a striped block group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message