hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gautam Borah <gautam.bo...@gmail.com>
Subject optimal size for Hbase.hregion.memstore.flush.size and its impact
Date Wed, 27 May 2015 20:53:12 GMT
Hi all,

The default size of Hbase.hregion.memstore.flush.size is define as 128 MB for Hbase.hregion.memstore.flush.size.
Could anyone kindly explain what would be the impact if we increase this to a higher value
512 MB or 800 MB or higher.

We have a very write heavy cluster. Also we run periodic end point co processor based jobs
that operate on the data written in the last 10-15 mins, every 10 minute. We are trying to
manage the memstore flush operations such that the hot data remains in memstore for at least
30-40 mins or longer, so that the job hits disk every 3rd or 4th time it tries to operate
on the hot data (it does scan). 

We have region server heap size of 20 GB and set the,

hbase.regionserver.global.memstore.lowerLimit = .45
hbase.regionserver.global.memstore.upperLimit = .55

We observed that if we set the Hbase.hregion.memstore.flush.size=128MB only 10% of the heap
is utilized by memstore, after that memstore flushes.

At Hbase.hregion.memstore.flush.size=512MB, we are able to increase the heap utelization to
by memstore to 35%. 

It would be very helpful for us to understand the implication of higher Hbase.hregion.memstore.flush.size
 for a long running cluster. 

Thanks,
Gautam
Mime
View raw message