hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5633) Improve OfflineImageViewer to use less memory
Date Fri, 06 Dec 2013 11:01:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841180#comment-13841180

Hudson commented on HDFS-5633:

FAILURE: Integrated in Hadoop-Yarn-trunk #413 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/413/])
HDFS-5633. Improve OfflineImageViewer to use less memory. Contributed by Jing Zhao. (jing9:
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionVisitor.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java

> Improve OfflineImageViewer to use less memory
> ---------------------------------------------
>                 Key: HDFS-5633
>                 URL: https://issues.apache.org/jira/browse/HDFS-5633
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>            Priority: Minor
>             Fix For: 2.4.0
>         Attachments: HDFS-5633.000.patch
> Currently after we rename a file/dir which is included in a snapshot, the file/dir can
be linked with two different reference INodes. To avoid saving/loading the inode multiple
times in/from FSImage, we use a temporary map to record whether we have visited this inode
> However, in OfflineImageViewer (specifically, in ImageLoaderCurrent), the current implementation
simply records all the directory inodes. This can take a lot of memory when the fsimage is
big. We should only record an inode in the temp map when it is referenced by an INodeReference,
just like what we do in FSImageFormat.

This message was sent by Atlassian JIRA

View raw message