hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
Date Thu, 29 Jan 2015 04:50:36 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296370#comment-14296370
] 

Kai Zheng commented on HADOOP-11514:
------------------------------------

Hi [~vinayrpet],

Thanks for your view.
bq.In fact whole ECChunk class itself is unnecessary in Common, as this is specific to 'striped'
design of the erasure encoding, but current Jira targets concrete APIs only. ECChunk can be
converted 'ByteBuffer' before passing to encode()/decode() of RawEncoder/RawDecoder in higher
levels.
As covered in HDFS-7662, the higher level construct *ErasureCoder* will mainly use ECChunk
type to call the raw coder, and how to converting ECChunk to bytes or ByteBuffer is better
to be located in the central place as I put in the base. That's why I thought *RawErasureCoder*
better support it in the API level. We're considering to place the whole erasure codec framework
in Common side, including the *ErasureCodec* API to be refined in HDFS-7699 and *ErasureCoder*
API defined in HDFS-7662. I understand there will be some questions as Zhe has already raised
in HDFS-7662, I guess we can discuss this there. To ease the discussion and commit of this,
I would remove the ECChunk related codes from the updated patch first and then put them back
in the patch for HDFS-7662. Hope this works.

> Raw Erasure Coder API for concrete encoding and decoding
> --------------------------------------------------------
>
>                 Key: HADOOP-11514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11514
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: HDFS-EC
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>         Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch,
HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch
>
>
> This is to abstract and define raw erasure coder API across different codes algorithms
like RS, XOR and etc. Such API can be implemented by utilizing various library support, such
as Intel ISA library and Jerasure library.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message