Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 27 Mar 2013 13:13:18 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12638146.1363832302935.63212.1364389998177@arcas>
In-Reply-To: <JIRA.12638146.1363832302935@arcas>
References: <JIRA.12638146.1363832302935@arcas>
Subject: [jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement
 for JAVA GC when using MSLAB
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HBASE-8163?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1361=
5214#comment-13615214 ]=20

Hudson commented on HBASE-8163:
-------------------------------

Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job=
/hbase-0.95-on-hadoop2/45/])
    HBASE-8163 MemStoreChunkPool: An improvement for JAVA GC when using MSL=
AB (Revision 1461399)

     Result =3D FAILURE
zjushch :=20
Files :=20
* /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/r=
egionserver/MemStore.java
* /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/r=
egionserver/MemStoreChunkPool.java
* /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/r=
egionserver/MemStoreFlusher.java
* /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/r=
egionserver/MemStoreLAB.java
* /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/r=
egionserver/TestMemStoreChunkPool.java

               =20
> MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
> --------------------------------------------------------------
>
>                 Key: HBASE-8163
>                 URL: https://issues.apache.org/jira/browse/HBASE-8163
>             Project: HBase
>          Issue Type: New Feature
>          Components: regionserver
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>             Fix For: 0.95.0, 0.98.0
>
>         Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch, hbase-8=
163v2.patch, hbase-8163v3.patch, hbase-8163v4.patch, hbase-8163v5.patch, hb=
ase-8163v6.patch
>
>
> *Usage:*
> Disable pool(default): configure "hbase.hregion.memstore.chunkpool.maxsiz=
e" as 0
> Enable pool: configure "hbase.hregion.memstore.chunkpool.maxsize" as a pe=
rcentage of global memstore size (between 0.0 and 1.0, recommend to set it =
as the gap between min and max sizes of memstore, e.g. 0.5)
> *Background*=EF=BC=9A
> When we use mslab,we will copy the keyvalue together in a structure calle=
d *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment.=20
> *Problem*=EF=BC=9A
> When one chunk is full, we will create a new chunk, and then the old chun=
k will be reclaimed by JVM if no reference to it.
> Mostly the chunk object would be promoted when doing Young GC, cause incr=
easing the cost of YGC=20
> When a Chunk object has no reference? It should meet two following condit=
ion:
> 1.Memstore which this chunk belongs to is flushed
> 2.No scanner is opening on the memstore which this chunk belongs to
> *Solution:*
> 1.Create a chunk pool to manage the no-reference chunks, instead of being=
 reclaimed by JVM
> 2.When a Chunk has no reference, put it back to the pool
> 3.The pool has a max capacity, it will skip the chunks when achieve the m=
ax size
> 4.When we need new Chunk to store KeyValue, get it from the pool if exist=
s, else create new one by pool, so we could be able to reuse the old chunks
> *Test results:*
> Environment:
> hbase-version:0.94
> -Xms4G -Xmx4G -Xmn2G
> Row size=3D50 bytes, Value size=3D1024 bytes
> 50 concurrent theads per client, insert 10,000,000 rows
> Before:
> Avg write request per second:12953
> After testing, final result of jstat -gcutil :
> YGC YGCT FGC FGCT GCT=20
> 747 36.503 48 2.492 38.995
> After:
> Avg write request per second:14025
> After testing, final result of jstat -gcutil :
> YGC YGCT FGC FGCT GCT=20
> 711 20.344 4 0.284 20.628
> *Improvement: YGC 40+%; WPS 5%+*
> review board :
> https://reviews.apache.org/r/10056/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira