hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7353) Raw Erasure Coder API for concrete encoding and decoding
Date Tue, 27 Jan 2015 03:24:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292919#comment-14292919

Kai Zheng commented on HDFS-7353:

Hi [~zhz] or [~szetszwo], 
1. I'm sorry to be about naming again, but regarding better name for dataSize, how about numDataUnits
or dataUnitsCount?
2. About why we need the 3rd version encode()/decode(), it is because in above layer in ErasureCoder,
ECChunks are extracted from blocks and then they're passed down here for the encoding/decoding.
How to get bytes or ByteBuffer from ECChunk, it depends and therefore better have the logic
centrally here. Generally, in pure Java implementation, bytes are allocated in heap and used;
in ISA-L, better to obtain ByteBuffer from off-heap for performance consideration.

> Raw Erasure Coder API for concrete encoding and decoding
> --------------------------------------------------------
>                 Key: HDFS-7353
>                 URL: https://issues.apache.org/jira/browse/HDFS-7353
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>             Fix For: HDFS-EC
>         Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch
> This is to abstract and define raw erasure coder API across different codes algorithms
like RS, XOR and etc. Such API can be implemented by utilizing various library support, such
as Intel ISA library and Jerasure library.

This message was sent by Atlassian JIRA

View raw message