Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 3 Oct 2013 07:28:54 +0000 (UTC)
From: "chunhui shen (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12624815.1356010648233.5571.1380785334575@arcas>
In-Reply-To: <JIRA.12624815.1356010648233@arcas>
References: <JIRA.12624815.1356010648233@arcas>
Subject: [jira] [Commented] (HBASE-7404) Bucket Cache:A solution about
 CMS,Heap Fragment and Big Cache on HBASE
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HBASE-7404?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1378=
4889#comment-13784889 ]=20

chunhui shen commented on HBASE-7404:
-------------------------------------

bq,For pure random gets I find that setting hbase.bucketcache.size =3D 0.4 =
is 60% slower than just setting hfile.block.cache.size =3D 0.4. Is that exp=
ected?

Maybe the usage of this feature is misunderstood.=20
It's not used to replace LruBlockCache. =20
Its function is reducing JVM GC under low cache hit ratio or creating a L2 =
cache for HBase

Thus, for the above test case where cache hit ratio is near 100%, it will h=
ave worse  performance than LruBlockCache. Becasue BucketCache will do a me=
mory copy when hitting the block

> Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
> ----------------------------------------------------------------------
>
>                 Key: HBASE-7404
>                 URL: https://issues.apache.org/jira/browse/HBASE-7404
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 0.94.3
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>             Fix For: 0.95.0
>
>         Attachments: 7404-0.94-fixed-lines.txt, 7404-trunk-v10.patch, 740=
4-trunk-v11.patch, 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v=
13.txt, 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, HBASE=
-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, hbase-7404-trunkv9.pat=
ch, Introduction of Bucket Cache.pdf
>
>
> First, thanks @neil from Fusion-IO share the source code.
> Usage:
> 1.Use bucket cache as main memory cache, configured as the following:
> =E2=80=93"hbase.bucketcache.ioengine" "heap"
> =E2=80=93"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a pe=
rcentage of max heap size)
> 2.Use bucket cache as a secondary cache, configured as the following:
> =E2=80=93"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The =
file path where to store the block data)
> =E2=80=93"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB=
, so 1024 means 1GB)
> =E2=80=93"hbase.bucketcache.combinedcache.enabled" false (default value b=
eing true)
> See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig=
 and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache
> What's Bucket Cache?=20
> It could greatly decrease CMS and heap fragment by GC
> It support a large cache space for High Read Performance by using high sp=
eed disk like Fusion-io
> 1.An implementation of block cache like LruBlockCache
> 2.Self manage blocks' storage position through Bucket Allocator
> 3.The cached blocks could be stored in the memory or file system
> 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCac=
he), combined with LruBlockCache to decrease CMS and fragment by GC.
> 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio=
 to store block) to enlarge cache space
> How about SlabCache?
> We have studied and test SlabCache first, but the result is bad, because:
> 1.SlabCache use SingleSizeCache, its use ratio of memory is low because k=
inds of block size, especially using DataBlockEncoding
> 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCach=
e and LruBlockCache, put the block to LruBlockCache again if hit in SlabCac=
he , it causes CMS and heap fragment don't get any better
> 3.Direct heap performance is not good as heap, and maybe cause OOM, so we=
 recommend using "heap" engine=20
> See more in the attachment and in the patch


--
This message was sent by Atlassian JIRA
(v6.1#6144)