hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lisheng Sun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
Date Mon, 01 Jul 2019 07:29:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875987#comment-16875987

Lisheng Sun commented on HDFS-13693:

Thank [~jojochuang] for your comments.

{quote}I am not a snapshot expert, but I'd be interested to see if INodeReference can be optimized
the same way. If you don't use snapshot and don't know it off the top of your head, I can
find a snapshotted fsimage and experiment it.
 Sorry, XiaoMi HDFS don't use snapshot and don't know it off the top of your head, unfortunately  I
can't find a snapshotted fsimage and experiment it.

  And I have added javadoc for the addChildAtLoading method and updated the patch.  Could
you have time to review this patch? Thank you.

> Remove unnecessary search in INodeDirectory.addChild during image loading
> -------------------------------------------------------------------------
>                 Key: HDFS-13693
>                 URL: https://issues.apache.org/jira/browse/HDFS-13693
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: zhouyingchao
>            Assignee: Lisheng Sun
>            Priority: Major
>         Attachments: HDFS-13693-001.patch
> In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added to their
parent INode's map one by one. The adding procedure will search a position in the parent's
map and then insert the child to the position. However, during image loading, the search is
unnecessary since the insert position should always be at the end of the map given the sequence
they are serialized on disk.
> Test this patch against a fsimage of a 70PB  cluster (200million files and 300million
blocks), the image loading time be reduced from 1210 seconds to 1138 seconds.So it can reduce
up to about 10% of time.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message