hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1366) reduce namenode startup time by optimising checkBlockInfo while loading fsimage
Date Thu, 02 Sep 2010 05:36:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905412#action_12905412
] 

dhruba borthakur commented on HDFS-1366:
----------------------------------------

{code}

"main" prio=10 tid=0x0000000040114800 nid=0x3f15 runnable [0x000000004022a000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.hadoop.hdfs.server.namenode.BlocksMap$BlockInfo.getNext(BlocksMap.java:277)
        at org.apache.hadoop.hdfs.util.LightWeightGSet.get(LightWeightGSet.java:121)
        at org.apache.hadoop.hdfs.server.namenode.BlocksMap.checkBlockInfo(BlocksMap.java:353)
        at org.apache.hadoop.hdfs.server.namenode.BlocksMap.addINode(BlocksMap.java:370)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addToParent(FSDirectory.java:319)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:956)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:813)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:370)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:141)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:328)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:305)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:254)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:332)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1120)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1129)

{code}

The idea here is that when the fsimage is being loaded, BlocksMap.checkBlockInfo() will never
find the block in the blocksMap, i.e. the call to blocks.get() will always return null.

> reduce namenode startup time by optimising checkBlockInfo while loading  fsimage 
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-1366
>                 URL: https://issues.apache.org/jira/browse/HDFS-1366
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>
> The namenode spends about 10 minutes reading in a 14 GB fsimage file into memory and
creating all the in-memory data structures. A jstack based debugger clearly shows that most
of the time during the fsimage load is spent in BlocksMap.checkBlockInfo. There is a easy
way to optimize this method especially for this code path.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message