hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin O'Dell" <ke...@rocana.com>
Subject Re: Compaction monitoring
Date Fri, 05 May 2017 14:07:53 GMT

  That is a great series of questions. What are you using for
instrumentation of your HBase cluster?  Cloudera Manager, Ambari, Ganglia,
Cacti, etc?  You are really asking a lot of performance based metric
questions. I don't think you will be able to answer your questions without
first being able to answer these questions:

Do you see the Major Compaction I/O/CPU/Memory spikes throughout the whole
"off-peak" window?

Do you have the host resources overhead to add additional compaction
threads to shorten it if so?

What do your responses times look like during your "off-peak hours" are you
still within your SLAs?

Answering these questions should quickly allow you to answer your first two
questions. Your last question is very interesting:

*how much degrades my performance if region size is becoming too large? <--
This is 100% depends, it depends on your environment, I/O usage, SLAs etc,
I am not sure if anyone has done documented compaction times based on
Region sizes.  You may have to do some trial and error here.

I hope this helps!

On Fri, May 5, 2017 at 8:47 AM, Alexander Ilyin <alexander@weborama.com>

> Hi,
> Tuning HBase performance I've found a lot of settings which affect
> compaction process (off-peak hours, time between compactions, compaction
> ratio, region sizes, etc.). They all seem to be useful and there are
> recommendations in the doc saying which values to set. But I found no way
> to assess how they actually affect my cluster performance, i.e. how much
> resources is taken by compaction and when. I would like to figure out which
> settings work best for my dataset and my specific workload but with only
> general recommendations in hand it seems difficult to do.
> For example, I have difficulties answering the following questions:
> * can I shorten my off-peak hours range?
> * can I afford to do compactions more often? or more aggressively?
> * how much degrades my performance if region size is becoming too large?
> HBase version I'm using is 1.1.2
> Alexander

Kevin O'Dell
Field Engineer
850-496-1298 | Kevin@rocana.com

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message