hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop John <anoop.hb...@gmail.com>
Subject Re: 答复: flushing + compactions after config change
Date Fri, 28 Jun 2013 12:08:36 GMT
     Basically when you increase the memstore flush size ( well ur aim
there is to reduce flushes and make data sit in memory for longer time) you
need to carefully consider the 2 things
1. What is the max heap and what is the % memory you have allocated max for
all the memstores in a RS. And see how many regions and CFs and whether the
allocated max memstore memory is enough. Basically you should not make
premature flushes because of global heap preassure. As you said this will
create many small HFiles and so compactions
2. Check on the max Hlogs setting. Becuase of too many hlogs also there can
be a forced flush which also can create small HFiles. The default value is
32. The HLOg rolling will happen when one HLog's size reaches 95% size of
ur HDFS block size. (This 95% is also configurable)  If you have increased
the flush size and heap allocated for memstores, still you can get
premature flushes because of this HLog stuff.

Also from 94 onwards there is  compression feature for HLog..  Have a look
at that also.

On Fri, Jun 28, 2013 at 6:52 AM, Viral Bajaria <viral.bajaria@gmail.com>wrote:

> Thanks for the detailed JVM OPTS. On a side note, I doubt the issue with
> too many hlogs has anything to do with that right ? Because I have enough
> spare heap space and my Memstore's are not at the lowerLimit of the total
> available heap (i.e. 0.35)
> -Viral
> On Thu, Jun 27, 2013 at 6:09 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:
> > Hope that helpful.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message