hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rashmi Vinayak (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7715) Implement the Hitchhiker erasure coding algorithm
Date Thu, 26 Mar 2015 21:34:56 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382723#comment-14382723

Rashmi Vinayak commented on HDFS-7715:

Hi Kai,

>So that means for all the three modes/versions of the algorithm, we can have flexible
parameters rather than hard-coded values, right.

Yes, all three modes of the algorithm can have flexible parameters without hard-coded values.

Re. using raw coders: By using raw coders, do you mean first encoding using RS completely
and then modify it for Hitchhiker? There will be some performance impact if done this way
since the encoder will have to process all the data twice: first to RS encode and then to
do modifications required for Hitchhiker. But as you pointed, for the sake of simplicity of
the implementation, it might be fine to try this approach and see how much is the performance

> Implement the Hitchhiker erasure coding algorithm
> -------------------------------------------------
>                 Key: HDFS-7715
>                 URL: https://issues.apache.org/jira/browse/HDFS-7715
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: jack liuquan
>         Attachments: HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch
> [Hitchhiker | http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf]
is a new erasure coding algorithm developed as a research project at UC Berkeley. It has been
shown to reduce network traffic and disk I/O by 25%-45% during data reconstruction. This JIRA
aims to introduce Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID. 

This message was sent by Atlassian JIRA

View raw message