hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11503) Integrate Chocolate Cloud RS coder implementation
Date Mon, 06 Mar 2017 19:48:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897919#comment-15897919

Andrew Wang commented on HDFS-11503:

>From [~sw0rdf1sh]:

Our understanding is that this is the intended way of shipping new native EC plugins. Some
glue code is always necessary in the framework (like with Liberasurecode). Isn't this the

Yes, for sure. Glue code is definitely fine. My concern though is that if CC's RS implementation
isn't bit-identical to ISA-L and the Java coder, then users won't be able to transparently
swap out the coder implementations. In this case, we'll need to add additional guardrails
in the form of new "erasure coding policies" which encapsulate the coder and its parameters
(e.g. CC-RS 6+3 with a 64KB cell size). We only have 5 supported policies right now, so adding
CC-RS would greatly increase this configuration space.

> Integrate Chocolate Cloud RS coder implementation
> -------------------------------------------------
>                 Key: HDFS-11503
>                 URL: https://issues.apache.org/jira/browse/HDFS-11503
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Andrew Wang
>            Assignee: Marcell Feher
> Quote from Marcell on HDFS-7285:
> First of all let me introduce ourselves: we are Chocolate Cloud from Denmark, we use
erasure coding to improve storage solutions. We already have Reed-Solomon and Random Linear
Network Coding backends for Liberasurecode, and now we are at the final stage of developing
our RS plugin to HDFS-EC. The performance of our plugin is similar to ISA-L's, in some configurations
we are better, in others we are worse (our initial speed comparison charts can be found here:
> We would like our plugin to become officially supported in Hadoop 3.0. We can already
provide a preliminary version of our (native) library and a patch with the necessary glue
code for the next alpha release.
> I'd like to know your thoughts about whether it's possible and how it could be achieved.
> P.S: I'm happy to share more details if there's interest

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message