hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
Date Mon, 28 Apr 2014 23:33:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983734#comment-13983734

Suresh Srinivas commented on HDFS-6293:

OfflineImageViewer just dumps the fsimage in a readable format. In the past given hierarchical
nature of the fsimage, the information printed was consumable. Now it is no longer so.

One solution we can do is - Add an option to print directory tree information (along the lines
ls -r) that works against fsimage. Given that the information is printed is no longer dependent
on fsimage structure itself, this can be backward compatible output (with the caveats tools
having to deal with extra information for newly added features such as ACLs). Once this is
in place, we can have backward compatibility expectations on that. What do you guys think?
We could also consider either building a tool that works efficiently in memory or reorganize
the fsimage to make that possible (hope we do not have to change fsimage, due to incompatibility

[~kihwal], can you please provide the use cases you are using OIV for?

> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>                 Key: HDFS-6293
>                 URL: https://issues.apache.org/jira/browse/HDFS-6293
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: Kihwal Lee
>            Priority: Blocker
>         Attachments: Heap Histogram.html
> There are issues with OIV when processing fsimages in protobuf. 
> Due to the internal layout changes introduced by the protobuf-based fsimage, OIV consumes
excessive amount of memory.  We have tested with a fsimage with about 140M files/directories.
The peak heap usage when processing this image in pre-protobuf (i.e. pre-2.4.0) format was
about 350MB.  After converting the image to the protobuf format on 2.4.0, OIV would OOM even
with 80GB of heap (max new size was 1GB).  It should be possible to process any image with
the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.  I also noticed
that the secret manager section has no tokens while there were unexpired tokens in the original
image (pre-2.4.0).  I did not check whether they were also missing in the new pb fsimage.

This message was sent by Atlassian JIRA

View raw message