hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Spiegelberg <nspiegelb...@fb.com>
Subject Re: major compaction and regionserver failure...
Date Sun, 11 Dec 2011 20:46:26 GMT
Andy,

Some detail on the current compaction algorithm.  Major compactions can be
triggered in 3 ways:

1) User requested : e.g. From the shell
2) Size based : compact when file <= sum(smaller_files) *
'hbase.hstore.compaction.ratio'

This is the normal minor compaction logic.  It will upgrade to a major
compaction if all files in the Store meet this criteria.

3) Time delayed : after (now - min(StoreFile.timestamp)) >
"hbase.hregion.majorcompaction" + rand() *
hbase.hregion.majorcompaction.jitter"

That's the config you're currently toggling.  Note that
"hbase.hregion.majorcompaction" is a relative toggle, not an absolute
toggle.  It's the relative time between major compactions.  Note that this
time will reset if a user-requested or size-based compaction is issued.
Also, jitter is enabled by default so that major compaction storms don't
occur.  

There is currently no way to control compaction on an absolute time (e.g.
every Sunday @ midnight).  Although this feature could be added, the
addition of multi-threaded compactions in 0.92 should eliminate the need
for this feature, because the major compaction queue will no longer slow
down the minor compaction queue.


On 12/11/11 11:40 AM, "Andy Sautins" <andy.sautins@returnpath.net> wrote:

>
>  Thanks Mikhail.  That's very helpful information.
>
>   We followed some of the advice that I believe was posted to this list
>in the past to schedule major compactions manually to run during off-peak
>times.  That has gone very well for us, except when failures happen it
>seems we end up not major compacting some regions.  We currently have
>hbase.hregion.majorcompaction set to 0 to disable automated major
>compactions.  I'm wondering if I manually run major compactions every 'x'
>days if I set hbase.hregion.majorcompaction to 'x+m' where m is some
>number of days after the number of scheduled compactions then ideally the
>automated major compaction would just catch any regions that didn't get
>compacted normally.  For example, if I want to major compact every
>Saturday then I could set hbase.hregion.majorcompaction to 9 days
>(777600000 milliseconds ) so if for some reason the weekly Saturday major
>compaction didn't succeed for a subset of the regions it would be caught
>by the automated compaction 2 days later.  I would assume that if the
>manually scheduled major compactions completed successfully the automated
>major compactions wouldn't kick in because not enough time had passed.
>
>    Curious if you think that is a reasonably way to manually schedule
>major compactions but also have a catch-all in case of regionserver
>failures.
>
>    Thanks again for your input.  Very helpful.
>
>-----Original Message-----
>From: Mikhail Bautin [mailto:bautin.mailing.lists@gmail.com]
>Sent: Sunday, December 11, 2011 3:10 AM
>To: user@hbase.apache.org
>Subject: Re: major compaction and regionserver failure...
>
>Hi Andy,
>
>We don't persist manual compaction requests across server restarts, so if
>e.g. a major compaction was requested manually and the regionserver
>crashed, the new regionserver that picks up that region will not know
>about the major compaction request. For automatic compactions, however,
>compaction criteria are the same on the old and the new region server, so
>we would expect the new server to figure out that a compaction is
>necessary and start it again from the beginning if the old regionserver
>died in the middle of a compaction.
>
>Thanks,
>--Mikhail
>
>On Sat, Dec 10, 2011 at 2:25 PM, Andy Sautins
><andy.sautins@returnpath.net>wrote:
>
>>
>>  Mikhail,
>>
>>  Thanks for the response.  So to summarize compaction requests in the
>> compaction queue of the failed regionserver are lost and not picked up
>> when the regions are picked up by another regionserver.  So if we lose
>> a regionserver during a major compaction the regions that had not yet
>> been major compacted on the failed regionserver will not be major
>> compacted as part of that major compaction request, correct?
>>
>>  Thanks for the insight.
>>
>>   Andy
>>
>> -----Original Message-----
>> From: Mikhail Bautin [mailto:bautin.mailing.lists@gmail.com]
>> Sent: Saturday, December 10, 2011 1:50 PM
>> To: user@hbase.apache.org
>> Subject: Re: major compaction and regionserver failure...
>>
>> Hi Andy,
>>
>> Compaction queues are not persisted between regionserver restarts, and
>> the results of an incomplete compaction are discarded. Compactions
>> write into an HFile in a temporary location and only move it to the
>> region/CF directory in case of success (at least, this is how it works
>>in trunk).
>>
>> Thanks,
>> --Mikhail
>>
>> On Saturday, December 10, 2011, Andy Sautins wrote:
>>
>> >
>> >   I was wondering if someone could tell me what happens if a
>> > regionserver fails during major compaction.  Are the entries that
>> > were in the compaction queue for the failed regionserver migrated
>> > along with the regions to another server or are those requests for
>> > major compaction effectively lost for that run of major compaction?
>> > I guess what I'm really asking is do I still get a full major
>> > compaction of all regions or do I lose the major compactions of the
>> > regions that were in the queue on the failed regionserver.
>> >
>> >   Any insight would be much appreciated.   We are currently running
>>HBase
>> > 0.90.3-cdh3u1.
>> >
>> >
>> >
>>


Mime
View raw message