hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sheng Chen <chensheng2...@gmail.com>
Subject Re: Is there a way to increase the compaction queue size?
Date Thu, 07 Jul 2011 10:49:13 GMT
Thank you very much.

Actually I was trying to manage splitting manually, and to leave the
compaction automatical.
I really do not want to block my inserts by waiting for compactions. Is it
possible to keep minor compaction working in the background, like starting
a periodic job to do 'compact mytable' in the shell?

Regards,
Sean


2011/7/7 Zhoushuaifeng <zhoushuaifeng@huawei.com>

> hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> It's better not setting this to max_value. Setting this to a limited value
> will help hbase to control the insert speed, and have enough time to finish
> compact. Suggest to setting this to less than 20.
>
> hbase.hstore.max.filesize = 100G, do you mean hbase.hregion.max.filesize ?
> This may be too large, if a store file too large, it will take very long
> time to compact. I suggest to set this less than 8G.
>
> And the compaction queue size means how many regions need compaction, not
> the number of files to compact. So, you have 10 regions, all need
> compaction, and the compaction queue is 10.
>
>
> Zhou Shuaifeng(Frank)
>
> -----Original Message-----
> From: Sheng Chen [mailto:chensheng2010@gmail.com]
> Sent: Thursday, July 07, 2011 3:36 PM
> To: user@hbase.apache.org
> Subject: Re: Is there a way to increase the compaction queue size?
>
> Thank you for your reply.
>
> No more minor compaction is going on, while there are still about 100
> HStoreFiles in each region.
> I think it might because the compactionQueueSize has a maximum of 10 for a
> region server (only one per region?), so after I stopped inserting and the
> 10 compactions in the queue were finished, no more compaction is triggered.
>
> I think that "hbase.hstore.compaction.max" sets the max number of files for
> one minor compaction, does it has anything to do with my problem?
>
> Thanks,
> Sean
>
> 2011/7/7 Stack <stack@duboce.net>
>
> > Try setting "hbase.hstore.compaction.max" to a value other than 10, its
> > default.
> >
> > What do you mean by 'no further minor compaction is queuing'?  There
> > is no more compaction going on? Are there exceptions in the log?
> >
> > St.Ack
> >
> > On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <chensheng2010@gmail.com>
> > wrote:
> > > Hi, all,
> > > I have a hbase region server of 10 regions and find the compaction
> queue
> > > size is always 10 at maximum.
> > > After I finished inserting 1 billion records, there are still hundreds
> of
> > > HFiles in each region and no further minor compaction is queuing.
> > >
> > > My settings,
> > > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> > > hbase.hstore.compactionThreshold=3
> > > hbase.hstore.max.filesize=100G
> > >
> > > Is it the expected behavior?
> > > Can I make the compaction queue larger in order to keep doing minor
> > > compaction in the background?
> > >
> > > Thank you.
> > >
> > > Sean
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message