Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <4D769BAC.70304@yagni.com>
Date: Tue, 08 Mar 2011 14:12:12 -0700
From: Wayne Conrad <wconrad@yagni.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
 rv:1.9.1.15) Gecko/20101030 Icedove/3.0.10
MIME-Version: 1.0
To: user@couchdb.apache.org
Subject: Re: Compaction Strategies
References: <4D6E9B72.7010804@databill.com> <4D75F22D.1090203@ianhobson.co.uk>
 <4D76248E.6080901@bclary.com>
In-Reply-To: <4D76248E.6080901@bclary.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 03/08/11 05:43, Bob Clary wrote:
> On 3/8/11 1:09 AM, Ian Hobson wrote:
>> After compacting, the database will have a given size on disk. Would it
>> be possible to test, and compact if this grew by (say) 15%?
>
> Wayne,
>
> You say that your database size is 0.6 TB. What is the change in size
> during the day? What is the change in size after the compaction? If your
> database is not increasing appreciably in size during the day and if the
> compacted database size is not appreciably smaller than the
> pre-compaction size, I don't think you are gaining much by compacting
> once per day. In fact, you are taking a significant performance hit if
> your compaction is running for 10 hours every day.
>
> Perhaps a simple change in compaction schedule from once per day to once
> per N days will help in the short term. Similarly to Robert's
> suggestion, I keep track of the initial size of the database as well as
> the initial sizes of each of the views and compact them whenever they
> double in size.
>
> Of course you will need to tailor the trigger point so as to not reach a
> point where you do not have sufficient disk space to complete the
> compaction.
>
> Until Bob Dionne's patch is released I think that is the best you can
> achieve.
>
> Unless compaction performance is significantly improved, you also need
> to consider the case where once your database grows to a sufficiently
> large size, your database will reach a state where it is always being
> compacted, i.e., it will take so long to complete compaction that it
> will need to be compacted again immediately after it finishes.
>
> /bc
>

Bob,

We backed off on how often we were triggering compaction when we noticed 
it taking 10 hours to do.  Once I'm doing sharding, I am going to use a 
growth strategy such as Iam Hobson suggests or the one you mention, 
though: A database that has not grown (much) is one that does not need 
compaction.

Today I started splitting the 0.6TB up into a few hundred individual 
databases.  Many of them get little traffic, so this will help 
significantly once the patch is available and I can compact only when 
there's space to be gained.  It will help even when using growth as a 
trigger.

Thanks,
Wayne Conrad