hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
Date Thu, 25 Feb 2016 19:57:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167762#comment-15167762

Jing Zhao commented on HDFS-8786:

bq. I will add an extra check against the BlockManager#corruptReplicas map. If true will trigger
EC reconstruction task otherwise replication task. 

corruptReplicas is not tracking if the whole blockInfo (i.e., the whole striped block group)
is corrupted or requires internal block reconstructions. It is only used to track corrupted
internal blocks/replicas that were reported by DataNode/client by detected through block reports.
Thus here the key is not to check corruptReplicas, but to check if we have enough healthy
internal blocks covering the complete ID range (i.e. {{hasAllInternalBlocks}}). I.e., we should
augment the current {{chooseSource4SimpleReplication}} method by adding choosing decommissioning
node logic.

For sorting the storages, we do not do real "sorting". Instead, we only need to check if we
have duplicated internal blocks in the block group, and if yes, we make sure a decommissioned
storage with duplicated block is put in the end. I.e., suppose we have storages
d0, d1, d2, d3, d4, d5, d6, d7, d8, d9
mapping to indices
0, 1, 2, 3, 4, 5, 6, 7, 8, 2

Thus we have duplicated internal blocks for b2, locating in d2 and d9. If we find that d2
is a decommissioned node and d9 is not, we should switch d2 and d9 in the storage list.

> Erasure coding: DataNode should transfer striped blocks before being decommissioned
> -----------------------------------------------------------------------------------
>                 Key: HDFS-8786
>                 URL: https://issues.apache.org/jira/browse/HDFS-8786
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Rakesh R
>         Attachments: HDFS-8786-001.patch, HDFS-8786-draft.patch
> Per [discussion | https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
under HDFS-8697, it's too expensive to reconstruct block groups for decomm purpose.

This message was sent by Atlassian JIRA

View raw message