Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 4 Jan 2017 01:20:58 +0000 (UTC)
From: "Gang Xie (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12774454.1423722130000.634517.1483492858586@Atlassian.JIRA>
In-Reply-To: <JIRA.12774454.1423722130000@Atlassian.JIRA>
References: <JIRA.12774454.1423722130000@Atlassian.JIRA> <JIRA.12774454.1423722130972@arcas>
Subject: [jira] [Commented] (HDFS-7784) load fsimage in parallel
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 04 Jan 2017 01:21:00 -0000


    [ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796797#comment-15796797 ] 

Gang Xie commented on HDFS-7784:
--------------------------------

The JVM setting:
      -Xmx102400m
      -Xms102400m
      -Xmn5508m
      -XX:MaxDirectMemorySize=3686m
      -XX:MaxPermSize=1024m
      -XX:+PrintGCApplicationStoppedTime
      -XX:+UseConcMarkSweepGC
      -verbose:gc
      -XX:+PrintGCDetails
      -XX:+PrintGCDateStamps
      -XX:SurvivorRatio=6
      -XX:+UseCMSCompactAtFullCollection
      -XX:CMSInitiatingOccupancyFraction=70
      -XX:+UseCMSInitiatingOccupancyOnly
      -XX:+CMSParallelRemarkEnabled
      -XX:+UseNUMA
      -XX:+CMSClassUnloadingEnabled
      -XX:CMSMaxAbortablePrecleanTime=10000
      -XX:TargetSurvivorRatio=80
      -XX:+UseGCLogFileRotation
      -XX:NumberOfGCLogFiles=100
      -XX:GCLogFileSize=128m
      -XX:CMSWaitDuration=8000
      -XX:+CMSScavengeBeforeRemark
      -XX:ConcGCThreads=16
      -XX:ParallelGCThreads=16
      -XX:+CMSConcurrentMTEnabled
      -XX:+SafepointTimeout
      -XX:MonitorBound=16384
      -XX:-UseBiasedLocking
      -XX:MaxTenuringThreshold=3
      -XX:+ParallelRefProcEnabled
      -XX:-OmitStackTraceInFastThrow

> load fsimage in parallel
> ------------------------
>
>                 Key: HDFS-7784
>                 URL: https://issues.apache.org/jira/browse/HDFS-7784
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Walter Su
>            Assignee: Walter Su
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-7784.001.patch, test-20150213.pdf
>
>
> When single Namenode has huge amount of files, without using federation, the startup/restart speed is slow. The fsimage loading step takes the most of the time. fsimage loading can seperate to two parts, deserialization and object construction(mostly map insertion). Deserialization takes the most of CPU time. So we can do deserialization in parallel, and add to hashmap in serial.  It will significantly reduce the NN start time.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org