hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-684) Use HAR filesystem to merge parity files
Date Tue, 12 Jan 2010 08:58:57 GMT

    [ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799125#action_12799125
] 

dhruba borthakur commented on HDFS-684:
---------------------------------------

Another issue is that the Har-ing of parity files should occur in a map-reduce job. Otherwise,
a single node will not be able to keep up with the rate of Har-ing all the party files in
a big cluster.

> Use HAR filesystem to merge parity files 
> -----------------------------------------
>
>                 Key: HDFS-684
>                 URL: https://issues.apache.org/jira/browse/HDFS-684
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: contrib/raid
>            Reporter: dhruba borthakur
>            Assignee: Rodrigo Schmidt
>         Attachments: HDFS-684.0.patch, HDFS-684.1.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is
RAIDed. This puts additional burden on the memory requirements of the namenode. It will be
 nice if the parity files are combined together using the HadoopArchive (har) format. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message