hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rashmi Vinayak (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7715) Implement the Hitchhiker erasure coding algorithm
Date Sun, 15 Mar 2015 06:57:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362258#comment-14362258
] 

Rashmi Vinayak commented on HDFS-7715:
--------------------------------------

Hi Jack,

Yes - since the RS code currently employed in HDFS-RAID does not have the 'all-XOR-parity'
property, we cannot employ Hitchhiker-XOR+ directly based on it. Hitchhiker-XOR+ and Hitchhiker-nonXOR
both have same savings in I/O and bandwidth and same reading patterns during reconstruction.
While the former can be used over only RS codes satisfying the 'all-XOR-parity' property,
the later can be used over any RS code. 

> Implement the Hitchhiker erasure coding algorithm
> -------------------------------------------------
>
>                 Key: HDFS-7715
>                 URL: https://issues.apache.org/jira/browse/HDFS-7715
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: jack liuquan
>
> [Hitchhiker | http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf]
is a new erasure coding algorithm developed as a research project at UC Berkeley. It has been
shown to reduce network traffic and disk I/O by 25%-45% during data reconstruction. This JIRA
aims to introduce Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message