hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7345) Local Repairable Codes (LRC)
Date Thu, 13 Nov 2014 08:48:34 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kai Zheng updated HDFS-7345:
----------------------------
    Description: HDFS-7285 proposes to support Erasure Coding inside HDFS, supports multiple
Erasure Coding codecs via pluggable codec framework (HDFS-7337) and implements Reed Solomon
code by default. This is to support a more advanced coding mechanism, Local Repairable Codes
(LRC). As discussed in [Xorbas Hadoop Project Page|http://smahesh.com/HadoopUSC/] and the
paper attached there, LRC can reduce the number of erasure coding chunks that need to be read
when recovering missing data chunks, while still keeping the storage overhead low.  (was:
HDFS-7285 proposes to support Erasure Coding inside HDFS, supports multiple Erasure Coding
codecs via pluggable framework and implements Reed Solomon code by default. This is to support
a more advanced coding mechanism, Local Reconstruction Codes (LRC). As discussed in the paper
(https://www.usenix.org/system/files/conference/atc12/atc12-final181_0.pdf), LRC reduces the
number of erasure coding fragments that need to be read when reconstructing data fragments
that are offline, while still keeping the storage overhead low. The important benefits of
LRC are that it reduces the bandwidth and I/Os required for repair reads over prior codes,
while still allowing a significant reduction in storage overhead. The implementation would
also consider how to distribute the calculating of local and global parity blocks to other
relevant DataNodes.)

> Local Repairable Codes (LRC)
> ----------------------------
>
>                 Key: HDFS-7345
>                 URL: https://issues.apache.org/jira/browse/HDFS-7345
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>
> HDFS-7285 proposes to support Erasure Coding inside HDFS, supports multiple Erasure Coding
codecs via pluggable codec framework (HDFS-7337) and implements Reed Solomon code by default.
This is to support a more advanced coding mechanism, Local Repairable Codes (LRC). As discussed
in [Xorbas Hadoop Project Page|http://smahesh.com/HadoopUSC/] and the paper attached there,
LRC can reduce the number of erasure coding chunks that need to be read when recovering missing
data chunks, while still keeping the storage overhead low.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message