hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload
Date Thu, 12 Dec 2013 03:59:15 GMT

     [ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Anoop Sam John updated HBASE-5349:
----------------------------------

    Release Note: 
Both memstore heap size and on heap block cache sizes can be tuned automatically within the
RS life time. The algorithm to be used for this tuning is also pluggable.
This feature adds the below new config
"hbase.regionserver.global.memstore.size.max.range" and "hbase.regionserver.global.memstore.size.min.range"
using which one can specify the total heap % within which the memstore size can vary.
"hfile.block.cache.size.max.range" and "hfile.block.cache.size.min.range" using which one
can specify the total heap % within which the block cache size can vary.
Using "hbase.regionserver.heapmemory.tuner.class" one can plugin an impl for the tuner algorithm.
Pass the FQCN of the tuner impl class which implements org.apache.hadoop.hbase.regionserver.HeapMemoryTuner
The period within which the tuner checks for a possible tune can be adjusted with "hbase.regionserver.heapmemory.tuner.period".
This defaults to 300000 (5 mins)
The tuner algorithm receives a TunerContext in which we pass the number of block cache evictions
(happened within that time) and the memstore flushes happened (Forced due to global heap pressure
and normal flushes as separate items)

Config changes
------------------
Changed the config name "hbase.regionserver.global.memstore.upperLimit" to "hbase.regionserver.global.memstore.size"
This will be the initial memstore size allocated (default value as 40%).
Also changed "hbase.regionserver.global.memstore.lowerLimit" to "hbase.regionserver.global.memstore.size.lower.limit"
There is a semantic change also here. "hbase.regionserver.global.memstore.lowerLimit" represented
the total heap % upon which we start force flushes for memstores and defaults to 38% of total
heap space.
The new one is the % of memstore max heap size (Represented by hbase.regionserver.global.memstore.size).
This defaults to 95% of hbase.regionserver.global.memstore.size value.


  was:
Both memstore heap size and on heap block cache sizes can be tuned automatically within the
RS life time. The algorithm to be used for this tuning is also pluggable.
This feature adds the below new config
"hbase.regionserver.global.memstore.size.max.range" and "hbase.regionserver.global.memstore.size.min.range"
using which one can specify the total heap % within which the memstore size can vary.
"hfile.block.cache.size.max.range" and "hfile.block.cache.size.max.range" using which one
can specify the total heap % within which the memstore size can vary.
Using "hbase.regionserver.heapmemory.tuner.class" one can plugin an impl for the tuner algorithm.
Pass the FQCN of the tuner impl class which implements org.apache.hadoop.hbase.regionserver.HeapMemoryTuner
The period within which the tuner checks for a possible tune can be adjusted with "hbase.regionserver.heapmemory.tuner.period".
This defaults to 300000 (5 mins)
The tuner algorithm receives a TunerContext in which we pass the number of block cache evictions
(happened within that time) and the memstore flushes happened (Forced due to global heap pressure
and normal flushes as separate items)

Config changes
------------------
Changed the config name "hbase.regionserver.global.memstore.upperLimit" to "hbase.regionserver.global.memstore.size"
This will be the initial memstore size allocated (default value as 40%).
Also changed "hbase.regionserver.global.memstore.lowerLimit" to "hbase.regionserver.global.memstore.size.lower.limit"
There is a semantic change also here. "hbase.regionserver.global.memstore.lowerLimit" represented
the total heap % upon which we start force flushes for memstores and defaults to 38% of total
heap space.
The new one is the % of memstore max heap size (Represented by hbase.regionserver.global.memstore.size).
This defaults to 95% of hbase.regionserver.global.memstore.size value.



> Automagically tweak global memstore and block cache sizes based on workload
> ---------------------------------------------------------------------------
>
>                 Key: HBASE-5349
>                 URL: https://issues.apache.org/jira/browse/HBASE-5349
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.92.0
>            Reporter: Jean-Daniel Cryans
>            Assignee: Anoop Sam John
>             Fix For: 0.99.0
>
>         Attachments: HBASE-5349_V2.patch, HBASE-5349_V3.patch, HBASE-5349_V4.patch, HBASE-5349_V5.patch,
WIP_HBASE-5349.patch
>
>
> Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores)
and Block Cache based on the workload. If you need an image, scroll down at the bottom of
this link: http://www.hypertable.com/documentation/architecture/
> That'd be one less thing to configure.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message