hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7344) Erasure Coding worker and support in DataNode
Date Wed, 25 Mar 2015 04:36:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379277#comment-14379277

Tsz Wo Nicholas Sze commented on HDFS-7344:

> In most recovery cases, each ECWorker only generates 1 block. ...

For 1 missing block, we may not need to recover it at all since (6,3)-Reed-Solomon can tolerate
3 missing blocks.  Also recovery is more efficient for 2- or  3- missing blocks.

> Erasure Coding worker and support in DataNode
> ---------------------------------------------
>                 Key: HDFS-7344
>                 URL: https://issues.apache.org/jira/browse/HDFS-7344
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Kai Zheng
>            Assignee: Li Bo
>         Attachments: HDFS ECWorker Design.pdf, hdfs-ec-datanode.0108.zip, hdfs-ec-datanode.0108.zip
> According to HDFS-7285 and the design, this handles DataNode side extension and related
support for Erasure Coding, and implements ECWorker. It mainly covers the following aspects,
and separate tasks may be opened to handle each of them.
> * Process encoding work, calculating parity blocks as specified in block groups and codec
> * Process decoding work, recovering data blocks according to block groups and codec schema;
> * Handle client requests for passive recovery blocks data and serving data on demand
while reconstructing;
> * Write parity blocks according to storage policy.

This message was sent by Atlassian JIRA

View raw message