Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of samar.opensource@gmail.com
 designates 209.85.210.43 as permitted sender)
Message-ID: <5134A467.4030700@gmail.com>
Date: Mon, 04 Mar 2013 19:10:55 +0530
From: "samar.opensource" <samar.opensource@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:17.0) Gecko/20130216 Thunderbird/17.0.3
MIME-Version: 1.0
To: user@hbase.apache.org
CC: Jean-Marc Spaggiari <jean-marc@spaggiari.org>
Subject: Re: Compaction time
References: <513442E3.2050605@gmail.com>
 <CALckxSMtr9NM=TV86KYebcbby9zXZzyNeNCV4eKXamv_W+-V5Q@mail.gmail.com>
 <51345404.4010909@gmail.com>
 <CAPQV63VeSN95QwFTKo-ZQQOPimwwazZ5vrs3OYtEC487pTivmQ@mail.gmail.com>
In-Reply-To: 
 <CAPQV63VeSN95QwFTKo-ZQQOPimwwazZ5vrs3OYtEC487pTivmQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------040103060307050402090405"

--------------040103060307050402090405
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Thanks Viral and Jean,
  Sounds like timing the compactions is the only options now. Is it ok 
we put a large number to *|hbase.hstore.blockingStoreFiles|* so that 
even if compaction happens, automatic or manual , blocking rarely happens.

Is compaction time like 20-40 mins normal for a 2GB store .

Regards,
Samar
On 04/03/13 6:20 PM, Jean-Marc Spaggiari wrote:
> So can you simply do something like disabling automatic compactions,
> and run small manual one after each light job? That way it might not
> impact you heavy job with a big 40 minutes compactions since data will
> almost alway be compacted correctly? You can even maybe keep the
> automatic compaction to on if you do that since when you heavy job
> will run, your data compaction will be almost totally done.
>
> JM
>
> 2013/3/4 samar.opensource <samar.opensource@gmail.com>:
>> Hi Viral,
>>    The jobs dont run often. may be few times a day(10 ). But the problem is
>> there are other application which are running other jobs which may not be as
>> heavy but running compaction any time might block those jobs.
>> so there is
>> heavy jobs running less frequently
>> and light jobs running more frequently
>>
>> Regards,
>> Samar
>>
>> On 04/03/13 12:26 PM, Viral Bajaria wrote:
>>> How often do you run those jobs ? Do they run periodically or are they
>>> running all the time ?
>>>
>>> If you have a predictable periodic behavior, you could disable automatic
>>> compaction and trigger it manually using a cron job (not the recommended
>>> approach, AFAIK). Or you could set the compaction to trigger at a set time
>>> of the day when you know your jobs are not running.
>>>
>>> -Viral
>>>
>>> On Sun, Mar 3, 2013 at 10:44 PM, samar.opensource <
>>> samar.opensource@gmail.com> wrote:
>>>
>>>> Hi,
>>>>      We are running some high load jobs which are mostly writes. During
>>>> these jobs, compaction is triggered which takes sometime as longs as
>>>> 40mins
>>>> to complete. This causes blocking (as others wait for compaction in the
>>>> queue). Please suggest how much compaction time is reasonable for
>>>> compacting 2Gb store files . And best way to avoid long blocking
>>>> compactions.
>>>> Using Cloudera hbase vesion 3u3.
>>>> Regards,
>>>> Samar
>>>>


--------------040103060307050402090405--