hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-6293) Issues with OIV processing PB-based fsimages
Date Mon, 28 Apr 2014 15:34:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983105#comment-13983105
] 

Kihwal Lee edited comment on HDFS-6293 at 4/28/14 3:33 PM:
-----------------------------------------------------------

Attaching heap histogram of OIV. The max heap was set to 2GB to make it go out of heap early
and dump the heap.  It only loaded  about 3M files/dirs before crashing. If we optimize the
PB inefficiencies, we might be able to make it work with 50% of the heap. But that will still
be too much.


was (Author: kihwal):
Attaching heap histogram of OIV. The max heap was set to 2GB to make it go out of heap early
and dump the heap.  I only loaded up about 3M files/dirs before crashing. If we optimize the
PB inefficiencies, we might be able to make it work with 50% of the heap. But that will still
be too much.

> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
>                 Key: HDFS-6293
>                 URL: https://issues.apache.org/jira/browse/HDFS-6293
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: Kihwal Lee
>            Priority: Blocker
>         Attachments: Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf. 
> Due to the internal layout changes introduced by the protobuf-based fsimage, OIV consumes
excessive amount of memory.  We have tested with a fsimage with about 140M files/directories.
The peak heap usage when processing this image in pre-protobuf (i.e. pre-2.4.0) format was
about 350MB.  After converting the image to the protobuf format on 2.4.0, OIV would OOM even
with 80GB of heap (max new size was 1GB).  It should be possible to process any image with
the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.  I also noticed
that the secret manager section has no tokens while there were unexpired tokens in the original
image (pre-2.4.0).  I did not check whether they were also missing in the new pb fsimage.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message