hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hanjie Gu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
Date Tue, 25 Apr 2017 08:00:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982530#comment-15982530
] 

Hanjie Gu edited comment on HBASE-7404 at 4/25/17 8:00 AM:
-----------------------------------------------------------

I have a question:
Why does each size in BucketSizeInfo have one more 1KB? 
such as below code in BucketAllocator.java:(5KB, ..., 17KB, ..., 65KB, ...)
```
  // Default block size is 64K, so we choose more sizes near 64K, you'd better
  // reset it according to your cluster's block size distribution
  // TODO Support the view of block size distribution statistics
  private static final int DEFAULT_BUCKET_SIZES[] = { 4 * 1024 + 1024, 8 * 1024 + 1024,
      16 * 1024 + 1024, 32 * 1024 + 1024, 40 * 1024 + 1024, 48 * 1024 + 1024,
      56 * 1024 + 1024, 64 * 1024 + 1024, 96 * 1024 + 1024, 128 * 1024 + 1024,
      192 * 1024 + 1024, 256 * 1024 + 1024, 384 * 1024 + 1024,
      512 * 1024 + 1024 };
```


was (Author: jackgu):
I have a question:
Why does each size in BucketSizeInfo have one more 1KB? 
such as below code:(5KB, ..., 17KB, ..., 65KB, ...)
```
  // Default block size is 64K, so we choose more sizes near 64K, you'd better
  // reset it according to your cluster's block size distribution
  // TODO Support the view of block size distribution statistics
  private static final int DEFAULT_BUCKET_SIZES[] = { 4 * 1024 + 1024, 8 * 1024 + 1024,
      16 * 1024 + 1024, 32 * 1024 + 1024, 40 * 1024 + 1024, 48 * 1024 + 1024,
      56 * 1024 + 1024, 64 * 1024 + 1024, 96 * 1024 + 1024, 128 * 1024 + 1024,
      192 * 1024 + 1024, 256 * 1024 + 1024, 384 * 1024 + 1024,
      512 * 1024 + 1024 };
```

> Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
> ----------------------------------------------------------------------
>
>                 Key: HBASE-7404
>                 URL: https://issues.apache.org/jira/browse/HBASE-7404
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 0.94.3
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>             Fix For: 0.95.0
>
>         Attachments: 7404-0.94-fixed-lines.txt, 7404-trunk-v10.patch, 7404-trunk-v11.patch,
7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, 7404-trunk-v14.patch, BucketCache.pdf,
hbase-7404-94v2.patch, HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, hbase-7404-trunkv9.patch,
Introduction of Bucket Cache.pdf
>
>
> First, thanks @neil from Fusion-IO share the source code.
> Usage:
> 1.Use bucket cache as main memory cache, configured as the following:
> –"hbase.bucketcache.ioengine" "heap" (or "offheap" if using offheap memory to cache
block )
> –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap
size)
> 2.Use bucket cache as a secondary cache, configured as the following:
> –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to
store the block data)
> –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB)
> –"hbase.bucketcache.combinedcache.enabled" false (default value being true)
> See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache
> What's Bucket Cache? 
> It could greatly decrease CMS and heap fragment by GC
> It support a large cache space for High Read Performance by using high speed disk like
Fusion-io
> 1.An implementation of block cache like LruBlockCache
> 2.Self manage blocks' storage position through Bucket Allocator
> 3.The cached blocks could be stored in the memory or file system
> 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined
with LruBlockCache to decrease CMS and fragment by GC.
> 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block)
to enlarge cache space
> How about SlabCache?
> We have studied and test SlabCache first, but the result is bad, because:
> 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block
size, especially using DataBlockEncoding
> 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache,
put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment
don't get any better
> 3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using
"heap" engine 
> See more in the attachment and in the patch



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message