hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jack liuquan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7715) Implement the Hitchhiker erasure coding algorithm
Date Wed, 25 Mar 2015 07:49:54 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379446#comment-14379446

jack liuquan commented on HDFS-7715:

Hi Kai,
Thanks for your comments!
1. the pom file doesn't need to change. I just modify it for building in my machine. sorry
for that.
2. I have refined my codes with your suggestions, coding style using google style.
3. current implementation codes of hh add some functions in GaloisField class for performance
improvement consideration.
RS and piggyback calculations are always together. We can separate RS and piggyback calculations
and use existing RS raw coders for hh, but need to sacrifice a little performance. I think
we should balance gains between using native RS raw coders and separate them.

for review:
1. I have only attach the code of hitchhiker-XOR version
2. Last time i use zip format cause i can't upload patch more than 30kb. 
Thanks again!

> Implement the Hitchhiker erasure coding algorithm
> -------------------------------------------------
>                 Key: HDFS-7715
>                 URL: https://issues.apache.org/jira/browse/HDFS-7715
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: jack liuquan
>         Attachments: HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch
> [Hitchhiker | http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf]
is a new erasure coding algorithm developed as a research project at UC Berkeley. It has been
shown to reduce network traffic and disk I/O by 25%-45% during data reconstruction. This JIRA
aims to introduce Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID. 

This message was sent by Atlassian JIRA

View raw message