hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcell Feher (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11503) Integrate Chocolate Cloud RS coder implementation
Date Tue, 07 Mar 2017 08:57:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898978#comment-15898978
] 

Marcell Feher edited comment on HDFS-11503 at 3/7/17 8:57 AM:
--------------------------------------------------------------

We are working on fixing compatibility issues with the existing Reed-Solomon codecs, it is
not an issue for this particular plugin. 

However, I see a more general problem here. There are several other erasure coding algorithms
like LRC whose integration should be possible by the HDFS-EC framework. (as a side-comment:
our company specializes in using Random Linear Network Coding for storage and we are exploring
the possibilities of an RLNC plugin to HDFS). 
My point: to make the system future-proof, I think the *framework shouldn't assume that codecs
are compatible with each other*. Fundamentally different erasure coding algorithms should
not be treated as an "exception" or "a special case which is not RS" that needs to be guarded
against. The ecosystem should be designed in a way where all codecs are treated as first class
citizens, regardless of what algorithm they implement. It's a nice feature to tell the user
which ones are byte-level compatible with each others, but this should not be a requirement.
Who knows how awesome new EC algorithms will be developed in the future.


was (Author: sw0rdf1sh):
We are working on fixing compatibility issues with the existing Reed-Solomon codecs, it is
not an issue for this particular plugin. 

However I see a more general problem here. There are several other erasure coding algorithms
like LRC whose integration should be possible by the HDFS-EC framework. (as a side-comment:
our company specializes in using Random Linear Network Coding for storage and we are exploring
the possibilities of an RLNC plugin to HDFS). 
My point: to make the system future-proof, I think the framework shouldn't assume that codecs
are compatible with each other. Fundamentally different erasure coding algorithms should not
be treated as an "exception" or "not RS" that needs to be guarded against. The ecosystem should
be designed in a way where all codecs are treated as first class citizens, regardless of what
algorithm they implement. It's a nice feature to tell the user which ones are byte-level compatible
with each others, but this should not be a requirement. Who knows how awesome new EC algorithms
will be developed in the future.

> Integrate Chocolate Cloud RS coder implementation
> -------------------------------------------------
>
>                 Key: HDFS-11503
>                 URL: https://issues.apache.org/jira/browse/HDFS-11503
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Andrew Wang
>            Assignee: Marcell Feher
>
> Quote from Marcell on HDFS-7285:
> First of all let me introduce ourselves: we are Chocolate Cloud from Denmark, we use
erasure coding to improve storage solutions. We already have Reed-Solomon and Random Linear
Network Coding backends for Liberasurecode, and now we are at the final stage of developing
our RS plugin to HDFS-EC. The performance of our plugin is similar to ISA-L's, in some configurations
we are better, in others we are worse (our initial speed comparison charts can be found here:
https://www.chocolate-cloud.cc/Plugins/HDFS-EC/hdfs.html).
> We would like our plugin to become officially supported in Hadoop 3.0. We can already
provide a preliminary version of our (native) library and a patch with the necessary glue
code for the next alpha release.
> I'd like to know your thoughts about whether it's possible and how it could be achieved.
> P.S: I'm happy to share more details if there's interest



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message