hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhoushuaifeng <zhoushuaif...@huawei.com>
Subject RE: Is there a way to increase the compaction queue size?
Date Thu, 07 Jul 2011 08:58:59 GMT
It's better not setting this to max_value. Setting this to a limited value will help hbase
to control the insert speed, and have enough time to finish compact. Suggest to setting this
to less than 20.

hbase.hstore.max.filesize = 100G, do you mean hbase.hregion.max.filesize ?
This may be too large, if a store file too large, it will take very long time to compact.
I suggest to set this less than 8G.

And the compaction queue size means how many regions need compaction, not the number of files
to compact. So, you have 10 regions, all need compaction, and the compaction queue is 10.

Zhou Shuaifeng(Frank)

-----Original Message-----
From: Sheng Chen [mailto:chensheng2010@gmail.com] 
Sent: Thursday, July 07, 2011 3:36 PM
To: user@hbase.apache.org
Subject: Re: Is there a way to increase the compaction queue size?

Thank you for your reply.

No more minor compaction is going on, while there are still about 100
HStoreFiles in each region.
I think it might because the compactionQueueSize has a maximum of 10 for a
region server (only one per region?), so after I stopped inserting and the
10 compactions in the queue were finished, no more compaction is triggered.

I think that "hbase.hstore.compaction.max" sets the max number of files for
one minor compaction, does it has anything to do with my problem?


2011/7/7 Stack <stack@duboce.net>

> Try setting "hbase.hstore.compaction.max" to a value other than 10, its
> default.
> What do you mean by 'no further minor compaction is queuing'?  There
> is no more compaction going on? Are there exceptions in the log?
> St.Ack
> On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <chensheng2010@gmail.com>
> wrote:
> > Hi, all,
> > I have a hbase region server of 10 regions and find the compaction queue
> > size is always 10 at maximum.
> > After I finished inserting 1 billion records, there are still hundreds of
> > HFiles in each region and no further minor compaction is queuing.
> >
> > My settings,
> > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> > hbase.hstore.compactionThreshold=3
> > hbase.hstore.max.filesize=100G
> >
> > Is it the expected behavior?
> > Can I make the compaction queue larger in order to keep doing minor
> > compaction in the background?
> >
> > Thank you.
> >
> > Sean
> >

View raw message