hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-808) Implement something like PAR2 support?
Date Fri, 04 Dec 2009 17:02:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785999#action_12785999
] 

Eli Collins commented on HDFS-808:
----------------------------------

You could use the parity blocks in the erasure encoding layer (HDFS-503) to implement the
parity blocks in PAR2, though given that HDFS has multiple replicas why are parity blocks
better than the current recovery mechanism (which just uses a non-corrupt block rather than
reconstructing the corrupt block from parity)?

> Implement something like PAR2 support?
> --------------------------------------
>
>                 Key: HDFS-808
>                 URL: https://issues.apache.org/jira/browse/HDFS-808
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Allen Wittenauer
>            Priority: Minor
>
> We really need an Idea issue type, because I'm not sure if this is really viable. :)
 Just sort of thinking "out loud".
> I was thinking about how file recovery works on services like Usenet to fix data corruption
when chunks of files are missing.  I wonder how hard it would be to implement something like
PAR2 [ http://en.wikipedia.org/wiki/Parchive ] automatically for large files.  We'd have the
advantage of being able to do it in binary of course and could likely hide the details within
HDFS itself.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message