hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "samar.opensource" <samar.opensou...@gmail.com>
Subject Re: Compaction time
Date Mon, 04 Mar 2013 13:40:55 GMT
Thanks Viral and Jean,
  Sounds like timing the compactions is the only options now. Is it ok 
we put a large number to *|hbase.hstore.blockingStoreFiles|* so that 
even if compaction happens, automatic or manual , blocking rarely happens.

Is compaction time like 20-40 mins normal for a 2GB store .

Regards,
Samar
On 04/03/13 6:20 PM, Jean-Marc Spaggiari wrote:
> So can you simply do something like disabling automatic compactions,
> and run small manual one after each light job? That way it might not
> impact you heavy job with a big 40 minutes compactions since data will
> almost alway be compacted correctly? You can even maybe keep the
> automatic compaction to on if you do that since when you heavy job
> will run, your data compaction will be almost totally done.
>
> JM
>
> 2013/3/4 samar.opensource <samar.opensource@gmail.com>:
>> Hi Viral,
>>    The jobs dont run often. may be few times a day(10 ). But the problem is
>> there are other application which are running other jobs which may not be as
>> heavy but running compaction any time might block those jobs.
>> so there is
>> heavy jobs running less frequently
>> and light jobs running more frequently
>>
>> Regards,
>> Samar
>>
>> On 04/03/13 12:26 PM, Viral Bajaria wrote:
>>> How often do you run those jobs ? Do they run periodically or are they
>>> running all the time ?
>>>
>>> If you have a predictable periodic behavior, you could disable automatic
>>> compaction and trigger it manually using a cron job (not the recommended
>>> approach, AFAIK). Or you could set the compaction to trigger at a set time
>>> of the day when you know your jobs are not running.
>>>
>>> -Viral
>>>
>>> On Sun, Mar 3, 2013 at 10:44 PM, samar.opensource <
>>> samar.opensource@gmail.com> wrote:
>>>
>>>> Hi,
>>>>      We are running some high load jobs which are mostly writes. During
>>>> these jobs, compaction is triggered which takes sometime as longs as
>>>> 40mins
>>>> to complete. This causes blocking (as others wait for compaction in the
>>>> queue). Please suggest how much compaction time is reasonable for
>>>> compacting 2Gb store files . And best way to avoid long blocking
>>>> compactions.
>>>> Using Cloudera hbase vesion 3u3.
>>>> Regards,
>>>> Samar
>>>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message