Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Message-ID: <25619104.10031287586228262.JavaMail.jira@thor>
Date: Wed, 20 Oct 2010 10:50:28 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Subject: [jira] Commented: (HDFS-1435) Provide an option to store fsimage
 compressed
In-Reply-To: <7720269.489351285891053519.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922987#action_12922987 ] 

Aaron T. Myers commented on HDFS-1435:
--------------------------------------

Thanks for the clarification, Hairong.

Konstantin commented in the original OIV JIRA (HADOOP-5467) that it would be nice if we eliminated the code duplication stemming from effectively having two distinct FS image loaders. Had we done that, you wouldn't have needed to remember to make this change in another place. This work probably shouldn't be done as part of this JIRA, this problem that you hit just reminded me of that.

I've filed HDFS-1465 to address this problem.

> Provide an option to store fsimage compressed
> ---------------------------------------------
>
>                 Key: HDFS-1435
>                 URL: https://issues.apache.org/jira/browse/HDFS-1435
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: checkpoint-limitandcompress.patch, trunkImageCompress.patch, trunkImageCompress1.patch, trunkImageCompress2.patch, trunkImageCompress3.patch
>
>
> Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network bandwidth when secondary NN uploads a new fsimage to primary NN.
> If we could store fsimage compressed, the problem could be greatly alleviated.
> I plan to provide a new configuration hdfs.image.compressed with a default value of false. If it is set to be true, fsimage is stored as compressed.
> The fsimage will have a new layout with a new field "compressed" in its header, indicating if the namespace is stored compressed or not.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.