Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <1643009428.261111266136589287.JavaMail.jira@brutus.apache.org>
Date: Sun, 14 Feb 2010 08:36:29 +0000 (UTC)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Commented: (MAPREDUCE-1491) Use HAR filesystem to merge
 parity files
In-Reply-To: <1417798041.259731266122548052.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833528#action_12833528 ] 

dhruba borthakur commented on MAPREDUCE-1491:
---------------------------------------------

Code changes look good. I have one question.

Suppose a directory /dhruba has 10 files in it. All files initially have a replication factor of 3. Then the RaidNode creates a xxx_raid.har that replaces all the parity files. Now suppose, a user deletes the first file /dhruba. Now /dhruba has only 9 files. This patch will now delete the har file associated with /dhruba. At this point, all the 9 files in /dhruba are left with a replication factor of 2 only! Am I understanding this right? Of course, the har file will get recreated pretty oon, but for some amount of time (however small) there could be only two replicas of a block. If my understanding is correct, can we create another JIRA that could address this situation?

> Use HAR filesystem to merge parity files 
> -----------------------------------------
>
>                 Key: MAPREDUCE-1491
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1491
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>         Attachments: MAPREDUCE-1491.0.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be  nice if the parity files are combined together using the HadoopArchive (har) format.
> This was (HDFS-684) before, but raid migrated to MAPREDUCE.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.