hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-808) Implement something like PAR2 support?
Date Fri, 04 Dec 2009 17:48:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786026#action_12786026

Allen Wittenauer commented on HDFS-808:

I'm thinking about the situation where you have the complete file except one or two blocks
are completely missing (i.e., no replicas).  Using something like PAR2 you'd be able to reconstruct
the missing block completely.

> Implement something like PAR2 support?
> --------------------------------------
>                 Key: HDFS-808
>                 URL: https://issues.apache.org/jira/browse/HDFS-808
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Allen Wittenauer
>            Priority: Minor
> We really need an Idea issue type, because I'm not sure if this is really viable. :)
 Just sort of thinking "out loud".
> I was thinking about how file recovery works on services like Usenet to fix data corruption
when chunks of files are missing.  I wonder how hard it would be to implement something like
PAR2 [ http://en.wikipedia.org/wiki/Parchive ] automatically for large files.  We'd have the
advantage of being able to do it in binary of course and could likely hide the details within
HDFS itself.  

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message