Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Sat, 17 Dec 2011 05:10:31 +0000 (UTC)
From: "Maheswaran Sathiamoorthy (Commented) (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: 
 <1300213046.22090.1324098631519.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <641231556.3362.1320517373076.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (MAPREDUCE-3361) Ability to use
 SimpleRegeratingCode to fix missing blocks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171463#comment-13171463 ] 

Maheswaran Sathiamoorthy commented on MAPREDUCE-3361:
-----------------------------------------------------

There is another way of doing it:
I will add a new erasure code type called SRC to ErasureCodeType (which has XOR, RS now) and start storing SRC coded files in /raidsrc (RS files stored in /raidrs, XOR in /raid). When a file corruption is detected 
and recoverBlockToFile is called, the first thing to do is to check whether the file is a parity file or a source file. By looking at the location it can be easily determined whether this is a parity file and if so which type. Now if its not a parity file, then it is a source file and we need to determine its corresponding parity file. This can be done by checking for a parity file first in /raidsrc, and then in /raidrs and /raid to find out where it is located. That way we can find the parity file too. 
The same thing can be done by determining the filesize, for which we still need to search for the parity file by going to /raidrs or /raid; so I think the above approach is a little bit cleaner. 
For reconstructing the file, in either approach, we need to pass the ErasureCodeType all the way till the decoder and encoder. 
                
> Ability to use SimpleRegeratingCode to fix missing blocks
> ---------------------------------------------------------
>
>                 Key: MAPREDUCE-3361
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3361
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/raid
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> ReedSolomon encoding (n, k) has n storage nodes and can tolerate n-k failures. Regenerating a block needs to access k blocks. This is a problem when n and k are large. Instead, we can use simple regenerating codes (n, k, f) that does first does ReedSolomon (n,k) and then does XOR with f stripe size. Then, a single disk failure needs to access only f nodes and f can be very small.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira