hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1435) Provide an option to store fsimage compressed
Date Wed, 13 Oct 2010 21:24:37 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920770#action_12920770
] 

Hairong Kuang commented on HDFS-1435:
-------------------------------------

@Lu, compressing the fsimage has additional advantage of reducing disk I/O and as well as
networkbandwidth when writing to a remote copy. I like your proposed optimizations like limiting
transmission speed and not to download an fsimage if the one at primary NameNode is the same
as the one at secondary NameNode. Could you please contribute those back to the community?

@Doug, thanks for your feedback. Hope that we will get some time to work on the avro format
soon.

> Provide an option to store fsimage compressed
> ---------------------------------------------
>
>                 Key: HDFS-1435
>                 URL: https://issues.apache.org/jira/browse/HDFS-1435
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: trunkImageCompress.patch
>
>
> Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network bandwidth when
secondary NN uploads a new fsimage to primary NN.
> If we could store fsimage compressed, the problem could be greatly alleviated.
> I plan to provide a new configuration hdfs.image.compressed with a default value of false.
If it is set to be true, fsimage is stored as compressed.
> The fsimage will have a new layout with a new field "compressed" in its header, indicating
if the namespace is stored compressed or not.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message