couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wayne Conrad <wcon...@yagni.com>
Subject Re: Compaction Strategies
Date Tue, 08 Mar 2011 21:12:12 GMT
On 03/08/11 05:43, Bob Clary wrote:
> On 3/8/11 1:09 AM, Ian Hobson wrote:
>> After compacting, the database will have a given size on disk. Would it
>> be possible to test, and compact if this grew by (say) 15%?
>
> Wayne,
>
> You say that your database size is 0.6 TB. What is the change in size
> during the day? What is the change in size after the compaction? If your
> database is not increasing appreciably in size during the day and if the
> compacted database size is not appreciably smaller than the
> pre-compaction size, I don't think you are gaining much by compacting
> once per day. In fact, you are taking a significant performance hit if
> your compaction is running for 10 hours every day.
>
> Perhaps a simple change in compaction schedule from once per day to once
> per N days will help in the short term. Similarly to Robert's
> suggestion, I keep track of the initial size of the database as well as
> the initial sizes of each of the views and compact them whenever they
> double in size.
>
> Of course you will need to tailor the trigger point so as to not reach a
> point where you do not have sufficient disk space to complete the
> compaction.
>
> Until Bob Dionne's patch is released I think that is the best you can
> achieve.
>
> Unless compaction performance is significantly improved, you also need
> to consider the case where once your database grows to a sufficiently
> large size, your database will reach a state where it is always being
> compacted, i.e., it will take so long to complete compaction that it
> will need to be compacted again immediately after it finishes.
>
> /bc
>

Bob,

We backed off on how often we were triggering compaction when we noticed 
it taking 10 hours to do.  Once I'm doing sharding, I am going to use a 
growth strategy such as Iam Hobson suggests or the one you mention, 
though: A database that has not grown (much) is one that does not need 
compaction.

Today I started splitting the 0.6TB up into a few hundred individual 
databases.  Many of them get little traffic, so this will help 
significantly once the patch is available and I can compact only when 
there's space to be gained.  It will help even when using growth as a 
trigger.

Thanks,
Wayne Conrad

Mime
View raw message