hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17204) Make L2 off heap cache default ON
Date Sun, 13 Aug 2017 17:22:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124992#comment-16124992

Anoop Sam John commented on HBASE-17204:

I was analyzing ur suggestion on config options for the off heap sizes for read and write
paths. Whether this should be some % of total max off heap size for the process. As u know,
as of now both parts have configs using which one can specify the exact size to be used. (Not
in %)

Write side - hbase.regionserver.offheap.global.memstore.size - Size in MBs
Read side  - hbase.bucketcache.size  To be specified in MBs size. This is like this from day
1 (Before 11425). The config allows to specify size as % but that is % of the heap size (Xmx)
means it can be used with on heap BC only. For file mode and off heap mode it has to be in
exact MBs size (There is no sanity check for this in code. Also we need and better documentation)

As u said we can change these to allow % of Max Direct memory size.  But will that be too
complicated for a user is what my thinking now. Lets see all Off heap memory using parts in
1. Off heap BC size 
2. Off heap MSLAB size
3. Off heap RPC side buffer pool
4. Misc -  Netty needs it. Java NIO needs it(RPC side as well as file read side).

When one config the BC size and MSLAB size as % of Max Direct memory, they have to have a
better judging what extra memory is needed for this misc and buffer pool.  The Buffer pool
need is fixed as that will have a fixed number of constant sized buffers (def to 64 KB and
2x handler count BBs). The misc need is also a headache.  Instead they will be knowing the
size needs of each of these BC and MSLAB.(How many GBs for each)
We already allowed the off heap BC size to specified as exact MBs size. Seems many users using
it already. Will be a BC break and better we can avoid if possible.

So IMO we can continue with current way of exact size specify using the above 2 configs

> Make L2 off heap cache default ON
> ---------------------------------
>                 Key: HBASE-17204
>                 URL: https://issues.apache.org/jira/browse/HBASE-17204
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 2.0.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>            Priority: Blocker
>             Fix For: 2.0.0
> L2 cache can be used for data blocks.  By default it is off now. After HBASE-11425 work,
L2 off heap cache can equally perform with L1 on heap cache. On heavy loaded workload, this
can even out perform L1 cache.  Pls see recently published report by Alibaba.  Also this work
was backported by Rocketfuel and similar perf improvement report from them too.
> Let us turn L2 off heap cache ON. As it is off heap, we can have much larger sized L2
BC.  What should be the default size?

This message was sent by Atlassian JIRA

View raw message