Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of chensheng2010@gmail.com
 designates 209.85.212.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <30D4476CB62BAE4589FD646476C29CD2CE9538@SZXEML503-MBS.china.huawei.com>
References: 
 <CAPz8+KBjJqM4r0ZNs6-zo8nuFL6ixoOz92OQm3PMx-ZyCHWSeA@mail.gmail.com>
	<CADcMMgEow9eg9Vd12qqsY1kmYZKhGYMLvFqjO=MbFddb2wFywg@mail.gmail.com>
	<CAPz8+KB6ESPR6p15ACASCpbVFOixbZU9D+p2afEYHKr1w5Z++w@mail.gmail.com>
	<30D4476CB62BAE4589FD646476C29CD2CE9538@SZXEML503-MBS.china.huawei.com>
Date: Thu, 7 Jul 2011 18:49:13 +0800
Message-ID: 
 <CAPz8+KC+N8EdJ2kQKFESymn9r69aw-BXveEoAtoxk9TPm0rtwA@mail.gmail.com>
Subject: Re: Is there a way to increase the compaction queue size?
From: Sheng Chen <chensheng2010@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf307f37cebb232904a77877e6

--20cf307f37cebb232904a77877e6
Content-Type: text/plain; charset=ISO-8859-1

Thank you very much.

Actually I was trying to manage splitting manually, and to leave the
compaction automatical.
I really do not want to block my inserts by waiting for compactions. Is it
possible to keep minor compaction working in the background, like starting
a periodic job to do 'compact mytable' in the shell?

Regards,
Sean


2011/7/7 Zhoushuaifeng <zhoushuaifeng@huawei.com>

> hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> It's better not setting this to max_value. Setting this to a limited value
> will help hbase to control the insert speed, and have enough time to finish
> compact. Suggest to setting this to less than 20.
>
> hbase.hstore.max.filesize = 100G, do you mean hbase.hregion.max.filesize ?
> This may be too large, if a store file too large, it will take very long
> time to compact. I suggest to set this less than 8G.
>
> And the compaction queue size means how many regions need compaction, not
> the number of files to compact. So, you have 10 regions, all need
> compaction, and the compaction queue is 10.
>
>
> Zhou Shuaifeng(Frank)
>
> -----Original Message-----
> From: Sheng Chen [mailto:chensheng2010@gmail.com]
> Sent: Thursday, July 07, 2011 3:36 PM
> To: user@hbase.apache.org
> Subject: Re: Is there a way to increase the compaction queue size?
>
> Thank you for your reply.
>
> No more minor compaction is going on, while there are still about 100
> HStoreFiles in each region.
> I think it might because the compactionQueueSize has a maximum of 10 for a
> region server (only one per region?), so after I stopped inserting and the
> 10 compactions in the queue were finished, no more compaction is triggered.
>
> I think that "hbase.hstore.compaction.max" sets the max number of files for
> one minor compaction, does it has anything to do with my problem?
>
> Thanks,
> Sean
>
> 2011/7/7 Stack <stack@duboce.net>
>
> > Try setting "hbase.hstore.compaction.max" to a value other than 10, its
> > default.
> >
> > What do you mean by 'no further minor compaction is queuing'?  There
> > is no more compaction going on? Are there exceptions in the log?
> >
> > St.Ack
> >
> > On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <chensheng2010@gmail.com>
> > wrote:
> > > Hi, all,
> > > I have a hbase region server of 10 regions and find the compaction
> queue
> > > size is always 10 at maximum.
> > > After I finished inserting 1 billion records, there are still hundreds
> of
> > > HFiles in each region and no further minor compaction is queuing.
> > >
> > > My settings,
> > > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> > > hbase.hstore.compactionThreshold=3
> > > hbase.hstore.max.filesize=100G
> > >
> > > Is it the expected behavior?
> > > Can I make the compaction queue larger in order to keep doing minor
> > > compaction in the background?
> > >
> > > Thank you.
> > >
> > > Sean
> > >
> >
>

--20cf307f37cebb232904a77877e6--