hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Too many hlogs
Date Mon, 02 Apr 2012 20:41:30 GMT
On Mon, Apr 2, 2012 at 12:27 PM, Miles Spielberg <miles@box.com> wrote:

>
> Our region server are each hosting ~270 regions. Our writes are extremely
> well distributed (our HBase keys are output from a hash function) and small
> (~100s of bytes). I believe that the writes are being so well distributed
> across the regions that no one MemStore grows large enough to be written to
> HDFS, forcing the RegionServer to retain the old WALs. When these flush
> storms happen, we also see a compaction storm as all of the StoreFiles are
> written out at once for the flushed regions, as well as increased latency.
>

Agreed.


> What would be the recommended way to deal with this situation?

Increase hbase.regionserver.maxlogs?


You'll end up with much longer log replay times when a server crashes, you
probably don't want that.


>

Decrease *hbase.hregion.memstore.flush.size?
>

Even if you decrease it enough so that you don't hit the too many hlogs
you'll still end up flushing tiny files which will trigger compactions a
lot too.


> Are there other configuration knobs to tweak to control how long MemStores
> stick around before being flushed?
>

The issue here is that you should only have the same amount of MemStore
space as you have HLogs. 270 regions (let's say they only have 1 family)
means 270 * 64MB = ~17GB of potential MemStore space but the default
maximum amount of data you can have in the HLogs is 32 * ~64MB = ~2GB (here
64MB the default block size, if you changed it for your HBase deployment
then swap in the value you're using).

So there's two problems here:

 - You are over-committing the MemStores, I'm pretty sure you don't even
dedicate 17GB to the region servers and the MemStores by default won't
occupy more than 40% of it.
 - You HLogs aren't tuned and can't hold all the MemStore data without
force flushing regions.

Pretty much the only solution here is to have less MemStores so less
regions. Also see:
http://hbase.apache.org/book/important_configurations.html#bigger.regions

J-D

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message