hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10403) Simplify offheap cache configuration
Date Wed, 30 Apr 2014 06:27:17 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985218#comment-13985218

stack commented on HBASE-10403:

+1 on notion; especially stuff like a percentage being applied to the XX:MaxDirectMemorySize
for BC use.

We'll need to make sure the old configs continue to work after instituting these new ones.

On there being less options, I was going to suggest just drop slab cache? That'd be one less
option.  i've not looked closely at it but going by Nick blog post numbers, it does more work
to get to same place at least when using DoubleBlockCache...and for CombinedBlockCache, BucketCache
has more options so lets just concentrate on making it work and easy to use and just let go
of SlabCache?  On the other hand, it is looking like there'll be some more BC' implementations
to play with before there will be less (HBASE-8894 and if we get around to acting on [~vrodionov]
suggestions regards cache) so we should just deal but let us let go of options that have been

Agree w/ [~zjushch] comments that exposing CBC and DBC will only confuse (Strike DBC since
you can set up BucketCache to do L1/L2 as DBC does?).

Maybe have configs:


... or something like that.

Thinking on it, even simpler is just have hbase.block.cache.impl and pass LRUBC, or BucketCache,
or CBC (they all implement BlockCache interface?)  Add a new class if you want to do a new
combination.  Keeps it simple.

On the patch:

Looks good. I like the refactorings.  We should refactor that setVictim thing... its awkward.

+1 moving bucket cache configs into bucket cache class.

Restore the old param below:

-   * @param direct true if allocate direct buffer
+   * @param isDirect true if allocate isDirect buffer

(isDirect is a method name, not a param name)

+1 on moving all the sizing into the class the size applies to.

Patch is great.  Only thing to decide is what for configs?  We have to support the old.

> Simplify offheap cache configuration
> ------------------------------------
>                 Key: HBASE-10403
>                 URL: https://issues.apache.org/jira/browse/HBASE-10403
>             Project: HBase
>          Issue Type: Bug
>          Components: io
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>            Priority: Minor
>         Attachments: HBASE-10403.0.patch
> The BucketCache (HBASE-7404) is a very nice piece of functionality which is hidden behind
complex configuration. Enabling it currently requires manual calculation of L1 cache. It'd
be nice to make this easier to use and conform better with the existing heap management tools
we already have.
> Turning it on currently requires explicitly setting LruBlockCache (L1) instance size
and IOEngine (L2) size, making sure that L1 size isn't too big vs global memstore and total
heap. This is further confused by hbase.bucketcache.size accepting a percentage of total heap
OR explicit size in MB. Enabling SlabCache is slightly easier in that it just accepts whatever
LruBlockCache is provided.
> Turning on BucketCache using off-heap mode could look like:
> hbase-env.sh:
>   -Xmx5000m
>   -XX:MaxDirectMemorySize=15000m
> hbase-site.xml:
>  - hbase.regionserver.global.memstore.size = 0.7
>  - hbase.regionserver.onheap.blockcache.size = 0.1
>  - hbase.regionserver.blockcache.impl = BucketCache
>  - hbase.bucketcache.ioengine = offheap
> The result being a CombinedCache instance with 500m LruBlockCache + 15000m ByteBufferIOEngine
running in direct mode.
> This example does a couple things (mostly for the admin):
>  - knows NOT to enable SlabCache
>  - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/
>  - maintains the validity of HBaseConfiguration's existing check that global MemStore
+ LruBlockCache == 0.8
>  - maps "BucketCache" into meaning "a CombinedCache instance with these implementations
for L1 and L2."
>  - Figures out appropriate values for hbase.bucketcache.size and hbase.bucketcache.percentage.in.combinedcache

This message was sent by Atlassian JIRA

View raw message