hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gang Xie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7784) load fsimage in parallel
Date Mon, 26 Dec 2016 08:59:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15777928#comment-15777928
] 

Gang Xie commented on HDFS-7784:
--------------------------------

Hello,
Any update about this improvement? Loading huge image really takes time. And it seems that
this improvement is quite necessary.

> load fsimage in parallel
> ------------------------
>
>                 Key: HDFS-7784
>                 URL: https://issues.apache.org/jira/browse/HDFS-7784
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Walter Su
>            Assignee: Walter Su
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-7784.001.patch, test-20150213.pdf
>
>
> When single Namenode has huge amount of files, without using federation, the startup/restart
speed is slow. The fsimage loading step takes the most of the time. fsimage loading can seperate
to two parts, deserialization and object construction(mostly map insertion). Deserialization
takes the most of CPU time. So we can do deserialization in parallel, and add to hashmap in
serial.  It will significantly reduce the NN start time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message