hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Dimakis (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3361) Ability to use SimpleRegeratingCode to fix missing blocks
Date Thu, 01 Dec 2011 23:30:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161300#comment-13161300
] 

Alex Dimakis commented on MAPREDUCE-3361:
-----------------------------------------

I can see that backwards compatibility would be crucial for a deployed system. It is not always
clear how to find if a parity block is a simple parity or an RS parity just by counting since
the config files might have different number of simple parities (our default kept the total
number of parities to 4 by having two RS and two 6 degree XORs) to keep the same storage overhead
as a (14,10) Reed Solomon. 

I think a cleaner way to understand what each parity is, can be done through the meta data
file or the folder it is in (right now how do you distinguish simple XOR to RS parities)?

                
> Ability to use SimpleRegeratingCode to fix missing blocks
> ---------------------------------------------------------
>
>                 Key: MAPREDUCE-3361
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3361
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/raid
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> ReedSolomon encoding (n, k) has n storage nodes and can tolerate n-k failures. Regenerating
a block needs to access k blocks. This is a problem when n and k are large. Instead, we can
use simple regenerating codes (n, k, f) that does first does ReedSolomon (n,k) and then does
XOR with f stripe size. Then, a single disk failure needs to access only f nodes and f can
be very small.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message