hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1491) Use HAR filesystem to merge parity files
Date Sun, 14 Feb 2010 08:36:29 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833528#action_12833528

dhruba borthakur commented on MAPREDUCE-1491:

Code changes look good. I have one question.

Suppose a directory /dhruba has 10 files in it. All files initially have a replication factor
of 3. Then the RaidNode creates a xxx_raid.har that replaces all the parity files. Now suppose,
a user deletes the first file /dhruba. Now /dhruba has only 9 files. This patch will now delete
the har file associated with /dhruba. At this point, all the 9 files in /dhruba are left with
a replication factor of 2 only! Am I understanding this right? Of course, the har file will
get recreated pretty oon, but for some amount of time (however small) there could be only
two replicas of a block. If my understanding is correct, can we create another JIRA that could
address this situation?

> Use HAR filesystem to merge parity files 
> -----------------------------------------
>                 Key: MAPREDUCE-1491
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1491
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>         Attachments: MAPREDUCE-1491.0.patch
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is
RAIDed. This puts additional burden on the memory requirements of the namenode. It will be
 nice if the parity files are combined together using the HadoopArchive (har) format.
> This was (HDFS-684) before, but raid migrated to MAPREDUCE.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message