hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
Date Thu, 29 Jan 2015 03:14:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296323#comment-14296323

Kai Zheng commented on HADOOP-11514:

Hi [~anu],

Thanks for you question.
bq. how does it know if the ECBlock (or ECChunk) is valid ?
In high level constructs like ECManager, it will check EC blocks are healthy or not (like
lost at all due to DataNode failure or corrupt by block checksum detected), if some block
is found in question then the erasure codec framework will play in the work and this *RawErasureCoder*
will be involved. The bad block will be marked as isErasred being true and the codec/coder
will know which block in a ECGroup or BlockGroup is to be recovered, and then ECChunks will
be extracted for decoding. 

> Raw Erasure Coder API for concrete encoding and decoding
> --------------------------------------------------------
>                 Key: HADOOP-11514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11514
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: HDFS-EC
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>         Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch,
HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch
> This is to abstract and define raw erasure coder API across different codes algorithms
like RS, XOR and etc. Such API can be implemented by utilizing various library support, such
as Intel ISA library and Jerasure library.

This message was sent by Atlassian JIRA

View raw message