hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Compaction time
Date Mon, 04 Mar 2013 12:50:37 GMT
So can you simply do something like disabling automatic compactions,
and run small manual one after each light job? That way it might not
impact you heavy job with a big 40 minutes compactions since data will
almost alway be compacted correctly? You can even maybe keep the
automatic compaction to on if you do that since when you heavy job
will run, your data compaction will be almost totally done.


2013/3/4 samar.opensource <samar.opensource@gmail.com>:
> Hi Viral,
>   The jobs dont run often. may be few times a day(10 ). But the problem is
> there are other application which are running other jobs which may not be as
> heavy but running compaction any time might block those jobs.
> so there is
> heavy jobs running less frequently
> and light jobs running more frequently
> Regards,
> Samar
> On 04/03/13 12:26 PM, Viral Bajaria wrote:
>> How often do you run those jobs ? Do they run periodically or are they
>> running all the time ?
>> If you have a predictable periodic behavior, you could disable automatic
>> compaction and trigger it manually using a cron job (not the recommended
>> approach, AFAIK). Or you could set the compaction to trigger at a set time
>> of the day when you know your jobs are not running.
>> -Viral
>> On Sun, Mar 3, 2013 at 10:44 PM, samar.opensource <
>> samar.opensource@gmail.com> wrote:
>>> Hi,
>>>     We are running some high load jobs which are mostly writes. During
>>> these jobs, compaction is triggered which takes sometime as longs as
>>> 40mins
>>> to complete. This causes blocking (as others wait for compaction in the
>>> queue). Please suggest how much compaction time is reasonable for
>>> compacting 2Gb store files . And best way to avoid long blocking
>>> compactions.
>>> Using Cloudera hbase vesion 3u3.
>>> Regards,
>>> Samar

View raw message