hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhoushuaifeng <zhoushuaif...@huawei.com>
Subject RE: big compaction queue size
Date Tue, 06 Sep 2011 08:39:53 GMT
In hbase version 0.90.x, compact running only one thread, so if your input is too fast, compact
will be not able to finish in time.
This 3  parameters may help: 
hbase.hstore.compactionThreshold  default 3, increase it, e.g. 10
hbase.hstore.blockingStoreFiles                           e.g. 30
hbase.hstore.compaction.max					    e.g. 20

How many is suitable depends on the store file size flushed by regionserver, if it's too small,
you can increase the count more.
Even more, you may need to control the input speed.

Zhou Shuaifeng(Frank)

-----Original Message-----
From: Xu-Feng Mao [mailto:m9suns@gmail.com] 
Sent: Tuesday, September 06, 2011 4:16 PM
To: hbase-user@hadoop.apache.org; user@hbase.apache.org
Subject: big compaction queue size


We're running a 33-regionserver hbase cluster on top of cdh3u0 suites. On
average, we have 2400 regions hosted
on each regionserver. (hbase.hregion.max.filesize is 1.5GB, and we have
value size up to 4MB per object).

I check the log of regionserver, it seems like the compaction queue size is
about 1700, and every the compaction action
takes about 1 minute, and more over, most of the compaction are triggered to
a major one.

My question are,

1. Would this cause the performance degradation? It seems like "GET" action
in the interval that two minutes before/after
the compaction takes much longer time than usual. I thought the compaction
is a asynchronous operation.
2. Any issue would cause long-term compaction?
3. It seems like HBASE-1476 is going to implement multi-threaded compaction,
I guess it would help to reduce
the size of compaction queue.

Thanks and regards,

Mao Xu-Feng

View raw message