hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
Date Fri, 19 Feb 2016 01:36:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153501#comment-15153501

Kai Zheng commented on HADOOP-11828:

Hi Rashmi,

bq. When looking at HHXORErasureDecodingStep, I see that 'RSRawDecoder' is being used.
I don't see this but I'm sure HH coders should not couple with specific RS implementation.
It picks up the concrete implementation from configuration.
bq. Isn't 'RSRawDecoder' the older java implementation borrowed from Facebook's HDFS-RAID?
I was under the impression that RSRawDecoder is the older version and RSRawDecoder2 is the
newer version. Is it not so?
You're almost right. Yeah right now RSRawDecoder is the old (from HDFS-RAID) and RSRawDecoder2
is the new Java one (compatible with ISA-L coder). Very soon the two coders will be renamed,
ref. HADOOP-12808.

> Implement the Hitchhiker erasure coding algorithm
> -------------------------------------------------
>                 Key: HADOOP-11828
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11828
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 3.0.0
>            Reporter: Zhe Zhang
>            Assignee: jack liuquan
>             Fix For: 3.0.0
>         Attachments: 7715-hitchhikerXOR-v2-testcode.patch, 7715-hitchhikerXOR-v2.patch,
HADOOP-11828-hitchhikerXOR-V3.patch, HADOOP-11828-hitchhikerXOR-V4.patch, HADOOP-11828-hitchhikerXOR-V5.patch,
HADOOP-11828-hitchhikerXOR-V6.patch, HADOOP-11828-hitchhikerXOR-V7.patch, HADOOP-11828-v8.patch,
HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch
> [Hitchhiker | http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf]
is a new erasure coding algorithm developed as a research project at UC Berkeley. It has been
shown to reduce network traffic and disk I/O by 25%-45% during data reconstruction while retaining
the same storage capacity and failure tolerance capability as RS codes. This JIRA aims to
introduce Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID. 

This message was sent by Atlassian JIRA

View raw message