hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bochun Bai (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-2555) Improve CheckPoint node GC performance
Date Wed, 16 Nov 2011 09:58:53 GMT
Improve CheckPoint node GC performance

                 Key: HDFS-2555
                 URL: https://issues.apache.org/jira/browse/HDFS-2555
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: name-node
            Reporter: Bochun Bai

CheckPoint NameNode uses a lot of memory to merge the edits to the fsimage.
And the previous checkpoint data cannot easily be GCed.

It is running with jdk-sun-1.6 with different GC parameters, JProfiler calculated heap size
shown below:
  For 1MB sized fsimage, -XX:-UseConcMarkSweepGC, heap size increases 10MB every checking
  For 1MB sized fsimage, -XX:-UseParallelGC, heap size increases and back to 10M every checking
  For 1.2GB sized fsimage, both GC mode, heap size increases 2.5GB every checking period.

I have two workaround, both of them are tested work fine. But I want discuss more about the
CheckPoint design.
  Workaround 1: set the check period to a very long time, and restart the java process every
    Pro: solves full GC problem immediately.
    Con: Cannot merge edits when it increased suddenly.

  Workaround 2: also reset the INode tree structure when reset the root node.
    Pro: both CMS and Parallel mode works
    Con: This may be a JDK-specific problem.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message