hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject How to avoid frequent and blocking Memstore Flushes? (was: Re: About HBase Memstore Flushes)
Date Sat, 19 May 2012 02:05:36 GMT
I have a feeling Alex is raising an important issue, but maybe it's not getting attention because
it's tl;dr?

Andy Purtell just wrote something very related in a different thread:

> "The amount of heap alloted for memstore is fixed by configuration.

>  HBase maintains this global limit as part of a strategy to avoid out
>  of memory conditions. Therefore, as the number of regions grow, the
>  available space for each region's memstore shrinks proportionally. If
>  you have a heap sized too small for region hosting demand, then when
>  the number of regions gets up there, HBase will be flushing constantly
>  tiny files and compacting endlessly."

So isn't the above a problem for anyone using HBase?  More precisely, this part:
"...when the number of regions gets up there, HBase will be flushing constantly tiny files
and compacting endlessly."

If this is not a problem, how do people work around this?  Somehow keep the number of regions
mostly constant, or...?


Thanks!

Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 



>________________________________
> From: Alex Baranau <alex.baranov.v@gmail.com>
>To: hbase-user@hadoop.apache.org; user@hbase.apache.org 
>Sent: Wednesday, May 9, 2012 6:02 PM
>Subject: Re: About HBase Memstore Flushes
> 
>Should I may be create a JIRA issue for that?
>
>Alex Baranau
>------
>Sematext :: http://blog.sematext.com/
>
>On Tue, May 8, 2012 at 4:00 PM, Alex Baranau <alex.baranov.v@gmail.com>wrote:
>
>> Hi!
>>
>> Just trying to check that I understand things correctly about configuring
>> memstore flushes.
>>
>> Basically, there are two groups of configuraion properties (leaving out
>> region pre-close flushes):
>> 1. determines when flush should be triggered
>> 2. determines when flush should be triggered and updates should be blocked
>> during flushing
>>
>> 2nd one is for safety reasons: we don't want memstore to grow without a
>> limit, so we forbid writes unless memstore has "bearable" size. Also we
>> don't want flushed files to be too big. These properties are:
>> * hbase.regionserver.global.memstore.upperLimit &
>> hbase.regionserver.global.memstore.lowerLimit [1]   (1)
>> * hbase.hregion.memstore.block.multiplier [2]
>>
>> 1st group (sorry for reverse order) is about triggering "regular flushes".
>> As flushes can be performed without pausing updates, we want them to happen
>> before conditions for "blocking updates" flushes are met. The property for
>> configuring this is
>> * hbase.hregion.memstore.flush.size [3]
>> (* there are also open jira issues for per colfam settings)
>>
>> As we don't want to perform too frequent flushes, we want to keep this
>> option big enough to avoid that. At the same time we want to keep it small
>> enough so that it triggers flushing *before* the "blocking updates"
>> flushing is triggered. This configuration is per-region, while (1) is per
>> regionserver. So, if we had constant (more or less) number of regions per
>> regionserver, we could choose the value in a such way that it is not too
>> small, but small enough. However it is usual situation when regions number
>> assigned to regionserver varies a lot during cluster life. And we don't
>> want to adjust it over time (which requires RSs restarts).
>>
>> Does thinking above make sense to you? If yes, then here are the questions:
>>
>> A. is it a goal to have more or less constant regions number per
>> regionserver? Can anyone share their experience if that is achievable?
>> B. or should there be any config options for setting up triggering flushes
>> based on regionserver state (not just individual regions or stores)? E.g.:
>>     B.1 given setting X%, trigger flush of biggest memstore (or whatever
>> is logic for selecting memstore to flush) when memstore takes up X% of heap
>> (similar to (1), but triggers flushing when there's no need to block
>> updates yet)
>>     B.2 any other which takes into account regions number
>>
>> Thoughts?
>>
>> Alex Baranau
>> ------
>> Sematext :: http://blog.sematext.com/
>>
>> [1]
>>
>>   <property>
>>     <name>hbase.regionserver.global.memstore.upperLimit</name>
>>     <value>0.4</value>
>>     <description>Maximum size of all memstores in a region server before
>> new
>>       updates are blocked and flushes are forced. Defaults to 40% of heap
>>     </description>
>>   </property>
>>   <property>
>>     <name>hbase.regionserver.global.memstore.lowerLimit</name>
>>     <value>0.35</value>
>>     <description>When memstores are being forced to flush to make room in
>>       memory, keep flushing until we hit this mark. Defaults to 35% of
>> heap.
>>       This value equal to hbase.regionserver.global.memstore.upperLimit
>> causes
>>       the minimum possible flushing to occur when updates are blocked due
>> to
>>       memstore limiting.
>>     </description>
>>   </property>
>>
>> [2]
>>
>>   <property>
>>     <name>hbase.hregion.memstore.block.multiplier</name>
>>     <value>2</value>
>>     <description>
>>     Block updates if memstore has hbase.hregion.block.memstore
>>     time hbase.hregion.flush.size bytes.  Useful preventing
>>     runaway memstore during spikes in update traffic.  Without an
>>     upper-bound, memstore fills such that when it flushes the
>>     resultant flush files take a long time to compact or split, or
>>     worse, we OOME.
>>     </description>
>>   </property>
>>
>> [3]
>>
>>   <property>
>>     <name>hbase.hregion.memstore.flush.size</name>
>>     <value>134217728</value>
>>     <description>
>>     Memstore will be flushed to disk if size of the memstore
>>     exceeds this number of bytes.  Value is checked by a thread that runs
>>     every hbase.server.thread.wakefrequency.
>>     </description>
>>   </property>
>>
>
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message