hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7369) Erasure coding: distribute block recovery work to DataNode
Date Wed, 11 Mar 2015 23:10:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357774#comment-14357774
] 

Jing Zhao commented on HDFS-7369:
---------------------------------

Thanks for working on this, Zhe! The patch looks good to me. Some comments and thoughts:
# For a striped block, I think it will be better to use BlockInfo(Striped), instead of its
individual blocks, as the basic unit for recovery. E.g., suppose we lose 2 blocks for a 6+3
EC block. For recovery, I guess we want these two blocks are recovered in a single recovery
work instead of 2.
# As you mentioned in HDFS-7912, {{BlockManager}} and {{ReplicationMonitor}} never see individual
data/parity blocks currently. But it may be better to have a more strict type restriction
in {{UnderReplicatedBlocks}}, {{ReplicationWork}}, {{ErasureCodingWork}}, and {{computeRecoveryWorkForBlocks}}'s
parameter.
# Also because of #1, we may want to define a list of target DNs in {{BlockCodecInfo}}?

> Erasure coding: distribute block recovery work to DataNode
> ----------------------------------------------------------
>
>                 Key: HDFS-7369
>                 URL: https://issues.apache.org/jira/browse/HDFS-7369
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-7369-000-part1.patch, HDFS-7369-000-part2.patch, HDFS-7369-001.patch
>
>
> This JIRA updates NameNode to handle background / offline recovery of erasure coded blocks.
It includes 2 parts:
> # Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to appropriate
priority levels. 
> # Update {{ReplicationMonitor}} to distinguish block codec tasks and send a new DataNode
command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message