hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop John <anoop.hb...@gmail.com>
Subject Re: HBase Region Server Memory Configuration
Date Wed, 06 Apr 2016 10:31:24 GMT
So u have given the max heap size for the RS process via Cloudera
manager and u fear that that much is not taken by the process.  U may
need to check the logs in CDH manager and the RS to see whether it is
correctly set

So u r using bucket cache(BC) in offheap mode right.  In 1.0 version
there wont be any direct off heap space usage from HBase code other
than BC code.  (Later versions have it).  But the SocketChannel uses
off heap space to move the data to network.  When written bytes to
socket channel, it first copy it to off heap bytebuffers and then the
OS level calls to do the network transfers.  So ya the process as such
would need more offheap memory than the size of the bucket cache.

At least in 1.0 the perf of off heap Bucket cache is not same as that
of L1 cache.  When both are in place, by default, we will keep the
index blocks etc (non data) in L1 and data blocks in L2.  That would
be an ideal choice IMO.  The separation allows the L1 blocks to stay
there with out getting possibly evicted. These are meta information
for the data blocks.   The data blocks can stay in L2 cache.


On Wed, Apr 6, 2016 at 3:24 PM, Amit Shah <amits.84@gmail.com> wrote:
> Hello,
> I am trying to understand hbase's memory configuration. I am using hbase
> version 1.0 through cloudera (version 5.5.2). I need to benchmark query
> performance (through phoenix 4.6) on hbase with and without bucket cache.
> While reading about hbase memory configuration I have following doubts
> 1. I have configured java heap size on the region server as 3.5 GB on the
> cloudera manager console though the region server web ui (server metrics ->
> Memory tab) shows the max heap to be 1.7. hbase.block.cache.size is
> configured to be 0.6, hbase.regionserver.global.memstore.lowerLimit is
> configured to be 0.1 while hbase.regionserver.global.memstore.upperLimit is
> configured to be 0.2 so that the blockcache + memstore combination does not
> exceed 0.8.
> What could be the reason behind the difference shown on the region server
> web ui?
> 2. While configuring bucket cache, there are mainly two configs to set the
> size of the bucket cache -XX:MaxDirectMemorySize
> and hbase.bucketcache.size. It is recommended that XX:MaxDirectMemorySize
>  should be of a higher value than hbase.bucketcache.size. What are the
> hbase operations that use the offheap memory apart from bucket cache?
> 3. Is it worth benchmarking bucketcache in isolation i.e. without
> LruBlockCache - hbase.block.cache.size set to 0?
> Appreciate your inputs.
> Thanks,
> Amit

View raw message