hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
Date Thu, 25 Feb 2016 13:27:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167188#comment-15167188

Rakesh R commented on HDFS-8786:

Thanks [~jingzhao] for the useful comments.
bq. We can create replication task for an EC block only when we can make sure we have all
the internal blocks (the source nodes can be in decommissioned/decommissioning state). Otherwise
even if we have decommissioned/decommissioning datanodes, we should still trigger an EC reconstruction
I will add an extra check against the {{BlockManager#corruptReplicas}} map. If true will trigger
EC reconstruction task otherwise replication task. OK?
bq. For changes in BlockInfoStriped, instead of directly mixing decommission related logic
inside, it will be better to add a sorting logic in BlockManager just as what we do for contiguous
I think you are suggesting to provide a comparator similar to {{DFSUtil.DECOM_COMPARATOR}}.
Just adding a thought to understand more on this change -> I could see for a (m+k) block
group, (m+k) storage units are sorted. say (0,1,2,3,4). With the current behavior it will
add new storage unit at the end of the list (0,1,2,3,4,0) and later after the removal of decommission
node, the indices list will become (1,2,3,4,5,0). Will this break the sorted order behavior?
Please correct me if I'm missing anything. Thanks!

> Erasure coding: DataNode should transfer striped blocks before being decommissioned
> -----------------------------------------------------------------------------------
>                 Key: HDFS-8786
>                 URL: https://issues.apache.org/jira/browse/HDFS-8786
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Rakesh R
>         Attachments: HDFS-8786-001.patch, HDFS-8786-draft.patch
> Per [discussion | https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
under HDFS-8697, it's too expensive to reconstruct block groups for decomm purpose.

This message was sent by Atlassian JIRA

View raw message