hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Iulia Zidaru <iulia.zid...@1and1.ro>
Subject Re: Major compaction time
Date Thu, 10 Mar 2011 09:06:10 GMT
  Thank you for your answer. It is very useful. Unfortunately, our write 
pattern changes files across all regions and it is manually triggered 
every day. Is there a way to schedule, or prioritize some regions for 
major compaction? It would be nice if the regions with more changes are 
compacted first and also if we could set a percentage of affected data 
set. Is there an extension point or something?

Thank you,
Iulia



On 03/09/2011 08:25 PM, Stack wrote:
> You are running w/ the defaults where major compaction runs every day?
>   It depends on how your data is changing but at the extreme, a major
> compaction is a rewrite of the total data set.
>
> We'll usually suggest that folks turn off the auto-major compaction
> and instead run it manually at time of low loading -- evening or w/e
> dependent on traffic -- or that they manage it in some way; perhaps
> some tables need the major compaction less than others.
>
> As to how long it takes, it depends.  At the moment the compactor is a
> single thread per region so it could take a while though 18 hours
> seems like a long time.   We won't major compact files that have
> already been major compacted, so unless your write pattern adds files
> across all regions, the second major compaction should run to
> completion in a shorter time.
>
> St.Ack
>
>
> On Wed, Mar 9, 2011 at 9:01 AM, Iulia Zidaru<iulia.zidaru@1and1.ro>  wrote:
>>   Hi all,
>>
>> We have a cluster with 6 machines, 875 regions / machine and 2.75TB data.
>> The system receives a constant amount of inserts and deletes. The compaction
>> took about 18 h (we looked at **hbase.regionserver.compactionQueueSize
>> metric in Ganglia).
>> How long does a Major compaction take on your systems? How long it is
>> supposed to take? Could we improve this time?
>>
>> Thank you,
>> Iulia
>>
>>
>>
>> --
>> Iulia Zidaru
>> Java Developer
>>
>> 1&1 Internet AG - Bucharest/Romania - Web Components Romania
>> 18 Mircea Eliade St
>> Sect 1, Bucharest
>> RO Bucharest, 012015
>> iulia.zidaru@1and1.ro
>> 0040 31 223 9153
>>
>>
>>


-- 
Iulia Zidaru
Java Developer

1&1 Internet AG - Bucharest/Romania - Web Components Romania
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
iulia.zidaru@1and1.ro
0040 31 223 9153

  


Mime
View raw message