Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Mon, 28 Apr 2014 17:50:17 +0000 (UTC)
From: "Marcelo Vanzin (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12710933.1398695378984.192796.1398707417611@arcas>
In-Reply-To: <JIRA.12710933.1398695378984@arcas>
References: <JIRA.12710933.1398695378984@arcas>
Subject: [jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based
 fsimages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983288#comment-13983288 ] 

Marcelo Vanzin commented on HDFS-6293:
--------------------------------------

Hi Kihwal,

We have developed some code internally that mitigates (but does not eliminate) some of these problems. For an image with 140M entries it would need in the ballpark of 7-8GB of heap space, from my pencil-and-napkin calculations. Also, it does not generate entries in order like LsrPBImage does, and it's tailored for the use case of listing the contents of the file system (so it completely ignores things like snapshots).

(The reason it still requires a lot of memory is, as you note, that it needs to load information about all inodes in memory; our code is just a little smarter about what information it loads. I don't think it's possible to make it much better without changing the data in the fsimage itself.)

If people are ok with those limitations, we could clean up our code and post it as a patch.

> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
>                 Key: HDFS-6293
>                 URL: https://issues.apache.org/jira/browse/HDFS-6293
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: Kihwal Lee
>            Priority: Blocker
>         Attachments: Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf. 
> Due to the internal layout changes introduced by the protobuf-based fsimage, OIV consumes excessive amount of memory.  We have tested with a fsimage with about 140M files/directories. The peak heap usage when processing this image in pre-protobuf (i.e. pre-2.4.0) format was about 350MB.  After converting the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of heap (max new size was 1GB).  It should be possible to process any image with the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.  I also noticed that the secret manager section has no tokens while there were unexpired tokens in the original image (pre-2.4.0).  I did not check whether they were also missing in the new pb fsimage.


--
This message was sent by Atlassian JIRA
(v6.2#6252)