hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bochun Bai (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2555) Improve CheckPoint node GC performance
Date Thu, 17 Nov 2011 02:59:51 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151754#comment-13151754
] 

Bochun Bai commented on HDFS-2555:
----------------------------------

The heap size increasing causes 100% time of Full GC and the node is not functional.

Call System.gc is good for 1MB fsimage but not the 1.2GB one.
Even without a explicit System.gc, the 1MB one can be cleaned after 10-15 period checking.
But I think 10x memory requirement is not reasonable.

The rootDir referenced a huge INode tree in memory, JVM must find the exactly object and cleans
them in order. In my situation, it is 11 million of BlockInfo and 8 million of INodeFile.
De-reference them makes GC more easier.
                
> Improve CheckPoint node GC performance
> --------------------------------------
>
>                 Key: HDFS-2555
>                 URL: https://issues.apache.org/jira/browse/HDFS-2555
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Bochun Bai
>         Attachments: HDFS-2555-for-0.21.0.patch
>
>
> CheckPoint NameNode uses a lot of memory to merge the edits to the fsimage.
> And the previous checkpoint data cannot easily be GCed.
> It is running with jdk-sun-1.6 with different GC parameters, JProfiler calculated heap
size shown below:
>   For 1MB sized fsimage, -XX:-UseConcMarkSweepGC, heap size increases 10MB every checking
period.
>   For 1MB sized fsimage, -XX:-UseParallelGC, heap size increases and back to 10M every
checking period.
>   For 1.2GB sized fsimage, both GC mode, heap size increases 2.5GB every checking period.
> I have two workaround, both of them are tested work fine. But I want discuss more about
the CheckPoint design.
>   Workaround 1: set the check period to a very long time, and restart the java process
every hour. 
>     Pro: solves full GC problem immediately.
>     Con: Cannot merge edits when it increased suddenly.
>   Workaround 2: also reset the INode tree structure when reset the root node.
>     Pro: both CMS and Parallel mode works
>     Con: This may be a JDK-specific problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message