hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7912) Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks
Date Tue, 10 Mar 2015 17:44:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355294#comment-14355294
] 

Zhe Zhang commented on HDFS-7912:
---------------------------------

Thanks for the work Jing! I'm working on HDFS-7369 which is related to this thought.

I haven't looked {{PendingReplicationBlocks}} in detail. For {{UnderReplicatedBlocks}} I guess
we might be able to use the original interface? After all its main job is to quantify the
risk level of all blocks. Maybe we can use the existing {{add(Block block, int curReplicas,
int decomissionedReplicas, int expectedReplicas)}} method to express the risk level of a striped
block group? The current HDFS-7369 patch uses the following calculation based on this [reasoning|https://issues.apache.org/jira/browse/HDFS-7369?focusedCommentId=14318849&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14318849]:
{code}
-    short expectedReplication = bc.getBlockReplication();
+    short expectedReplication = bc.isStriped()?
+        HdfsConstants.NUM_PARITY_BLOCKS + 1 :
+        bc.getBlockReplication();
     NumberReplicas num = countNodes(block);
     int numCurrentReplica = num.liveReplicas();
+    if (bc.isStriped()) {
+      numCurrentReplica -= (HdfsConstants.NUM_DATA_BLOCKS - 1);
+      if (numCurrentReplica < 0) {
+        numCurrentReplica = 0;
+      }
+    }
{code}

Of course explicitly tracking {{BlockInfo}} gives even richer info to calculate the risk level,
and I don't see a downside except for the code change itself.

> Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7912
>                 URL: https://issues.apache.org/jira/browse/HDFS-7912
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HDFS-7912.000.patch
>
>
> Now with striped blocks and the design that uses a single BlockInfoStriped object to
track all the corresponding blocks, we need to clearly distinguish the type Block and BlockInfo
in BlockManager. Specifically, data structures like {{UnderReplicatedBlocks}} and {{PendingReplicationBlocks}}
should track BlockInfo instead of Block in order to support striped block recovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message