hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
Date Wed, 27 Mar 2013 13:13:18 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615214#comment-13615214
] 

Hudson commented on HBASE-8163:
-------------------------------

Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/])
    HBASE-8163 MemStoreChunkPool: An improvement for JAVA GC when using MSLAB (Revision 1461399)

     Result = FAILURE
zjushch : 
Files : 
* /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
* /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java
* /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
* /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java
* /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java

                
> MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
> --------------------------------------------------------------
>
>                 Key: HBASE-8163
>                 URL: https://issues.apache.org/jira/browse/HBASE-8163
>             Project: HBase
>          Issue Type: New Feature
>          Components: regionserver
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>             Fix For: 0.95.0, 0.98.0
>
>         Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch, hbase-8163v2.patch,
hbase-8163v3.patch, hbase-8163v4.patch, hbase-8163v5.patch, hbase-8163v6.patch
>
>
> *Usage:*
> Disable pool(default): configure "hbase.hregion.memstore.chunkpool.maxsize" as 0
> Enable pool: configure "hbase.hregion.memstore.chunkpool.maxsize" as a percentage of
global memstore size (between 0.0 and 1.0, recommend to set it as the gap between min and
max sizes of memstore, e.g. 0.5)
> *Background*:
> When we use mslab,we will copy the keyvalue together in a structure called *MemStoreLAB$Chunk*,
therefore we could decrease the heap fragment. 
> *Problem*:
> When one chunk is full, we will create a new chunk, and then the old chunk will be reclaimed
by JVM if no reference to it.
> Mostly the chunk object would be promoted when doing Young GC, cause increasing the cost
of YGC 
> When a Chunk object has no reference? It should meet two following condition:
> 1.Memstore which this chunk belongs to is flushed
> 2.No scanner is opening on the memstore which this chunk belongs to
> *Solution:*
> 1.Create a chunk pool to manage the no-reference chunks, instead of being reclaimed by
JVM
> 2.When a Chunk has no reference, put it back to the pool
> 3.The pool has a max capacity, it will skip the chunks when achieve the max size
> 4.When we need new Chunk to store KeyValue, get it from the pool if exists, else create
new one by pool, so we could be able to reuse the old chunks
> *Test results:*
> Environment:
> hbase-version:0.94
> -Xms4G -Xmx4G -Xmn2G
> Row size=50 bytes, Value size=1024 bytes
> 50 concurrent theads per client, insert 10,000,000 rows
> Before:
> Avg write request per second:12953
> After testing, final result of jstat -gcutil :
> YGC YGCT FGC FGCT GCT 
> 747 36.503 48 2.492 38.995
> After:
> Avg write request per second:14025
> After testing, final result of jstat -gcutil :
> YGC YGCT FGC FGCT GCT 
> 711 20.344 4 0.284 20.628
> *Improvement: YGC 40+%; WPS 5%+*
> review board :
> https://reviews.apache.org/r/10056/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message