Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of bcitrin@gmail.com designates
 209.85.220.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOxguM1HU-Cvu-s33BTcknF=x2kJ9iykFXTx3sJti32qLoRQKA@mail.gmail.com>
References: 
 <CAOxguM3Kt+fgyWUU794khL3UG2T=K3FJaf7Jo4pkCTTtCaDKHQ@mail.gmail.com>
	<CAHdjipLAV4hRQH4cR3kUuB36nGuukpc-wzwMD-c4d7Ug-N3LxA@mail.gmail.com>
	<CAOxguM1a8kiOmV9QnV64rrDx-39O_mPvpowFDCqT85ovoQjQ8w@mail.gmail.com>
	<CAHdjipJkOogUvKSypDiy-yHequCkpw2ZZ9y1+BGUMZqENKUQPA@mail.gmail.com>
	<CAOxguM1F9A-eui6gYZvm5gQ3ad_w+SPYMBie77dQvD-eySVtWw@mail.gmail.com>
	<CAOxguM1HU-Cvu-s33BTcknF=x2kJ9iykFXTx3sJti32qLoRQKA@mail.gmail.com>
Date: Mon, 4 Nov 2013 10:18:30 +0200
Message-ID: 
 <CAOxguM3SVZawMsV15nanuyo2fpKWEkHpp+a_WEGUuZ4kjXWPXQ@mail.gmail.com>
Subject: Re: Finding the right value for compaction configuration parameters
From: Boaz Citrin <bcitrin@gmail.com>
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=047d7b6731faaf417104ea558f2d

--047d7b6731faaf417104ea558f2d
Content-Type: text/plain; charset=ISO-8859-1

View compaction averages of database of 1.5M docs (same one used for the
previous tests) on a Dell Latitude laptop with Windows 7 64b OS, 8G RAM,
Intel Core i7-3720QM CPU (2 * 2.6 GHz):

keyvalue_buffer_size | average time (in minutes)
================================================
2097152              | 0:56
4194304              | 0:52
8388608              | 0:55
16777216            | 0:59
33554432            | 1:00
67108864            | 1:07
134217728          | 1:22
268435456          | 1:45

Same database, this time on a desktop, Windows Server 2008 R2, 2GB RAM,
Inter Xeon 3050 2*13 GHz:

2097152              | 2:30
4194304              | 2:11
8388608              | 2:18
16777216            | 2:33
33554432            | 2:55
67108864            | 4:15
134217728          | 5:43
268435456          | 6:42

Database has 3 views in different 3 design documents.

Would like to know some numbers from others, to get an idea of good default
values that can be assumed for various production environments.


On Sun, Nov 3, 2013 at 10:02 AM, Boaz Citrin <bcitrin@gmail.com> wrote:

> More numbers...
>
> Same database, this time on a desktop, Windows Server 2008 R2, 2GB RAM,
> Inter Xeon 3050 2*13 GHz.
>
>
> checkpoint_after |doc_buffer_size | avg. compaction time (in minutes)
> ====================================================================
> 41943040            | 4194304             | 17:26
> 83886080            | 8388608             | 12:01
> 167772160          | 16777216          | 9:34
> 335544320          | 33554432          | 8:56
> 671088640          | 67108864          | 9:26 (*)
> 1342177280        | 134217728       | 10:20
> 2684354560        | 268435456       | 12:28
>
> 671088640          | 4194304            | 17:25
> 671088640          | 8388608            | 12:17
> 671088640          | 16777216          | 10:15
> 671088640          | 33554432          | 8:56
> 671088640          | 67108864          | 9:20 (*)
> 671088640          | 134217728        | 10:16
> 671088640          | 268435456        | 12:19
>
>
>
>
>
> On Fri, Nov 1, 2013 at 5:44 PM, Boaz Citrin <bcitrin@gmail.com> wrote:
>
>> OK some numbers when running compaction of a database of 1.5M docs on a
>> Dell Latitude laptop with Windows 7 64b OS, 8G RAM, Intel Core i7-3720QM
>> CPU (2 * 2.6 GHz).
>>
>> checkpoint_after |doc_buffer_size | avg. compaction time (in minutes)
>> ======================================================
>> 41943040            | 4194304             | 6:11
>> 83886080            | 8388608             | 5:10
>> 167772160          | 16777216          | 4:49
>> 335544320          | 33554432          | 4:06
>> 671088640          | 67108864          | 3:34 (*)
>> 1342177280        | 134217728       | 4:16
>> 2684354560        | 268435456       | 5:19
>>
>> 671088640          | 4194304            | 5:39
>> 671088640          | 8388608            | 4:06
>> 671088640          | 16777216          | 3:44
>> 671088640          | 33554432          | 3:20
>> 671088640          | 67108864          | 3:16 (*)
>> 671088640          | 134217728        | 3:44
>> 671088640          | 268435456        | 4:28
>>
>> As you can see in the starred lines, same parameters might give different
>> results, but I think the direction is clear;
>> In my situation buffer of 64m gives the fastest compaction, and we also
>> see that as expected the checkpoint size affects the compaction time as
>> well.
>>
>> Note that this is NOT the time that an average daily compaction takes as
>> I ran these consecutively, i.e. the database was initially not fragmented.
>>
>>
>>
>> On Sat, Oct 26, 2013 at 5:25 PM, Alexander Shorin <kxepal@gmail.com>wrote:
>>
>>> Feel free to post here! I think others would be also interested in
>>> your experience and could share (I hope so) their own too.
>>> --
>>> ,,,^..^,,,
>>>
>>>
>>> On Sat, Oct 26, 2013 at 7:05 PM, Boaz Citrin <bcitrin@gmail.com> wrote:
>>> > Hi Alexander,
>>> >
>>> > Yes, I have some numbers, do you want me to share here or somewhere
>>> else?
>>> >
>>> > Best,
>>> >
>>> > Boaz
>>> >
>>> >
>>> > On Mon, Oct 21, 2013 at 12:32 AM, Alexander Shorin <kxepal@gmail.com>
>>> wrote:
>>> >
>>> >> Hi Boaz!
>>> >>
>>> >> On Fri, Oct 18, 2013 at 12:57 AM, Boaz Citrin <bcitrin@gmail.com>
>>> wrote:
>>> >> > Testing database compaction with various doc_buffer_size values I
>>> get
>>> >> > completely different results.
>>> >> > Documentation is fairly vague, so I wonder how to choose the right
>>> value;
>>> >> > What parameters affect this - HD buffer size, average doc size,
>>> >> > database size, fragmentation, etc... ?!
>>> >> >
>>> >> > Same goes for view compaction and keyvalue_buffer_size.
>>> >>
>>> >> These parameters does affect on what they are named: they defines
>>> >> buffer size for copying data from db/view file to the .compact one.
>>> >> What information you think is missed?
>>> >>
>>> >> > (For me the the compaction with default values was many times slower
>>> >> > than with the values that gave the faster compaction).
>>> >>
>>> >> All numbers have their cost: large buffers requires more memory while
>>> >> they reduces I/O operations and vice versa. Much likely, that default
>>> >> values wouldn't provide you high performance since they aimed to fit
>>> >> everyone, but that's why you may tweak them for your needs (:
>>> >>
>>> >> I believe, that it's possible to revise them, but first need to
>>> >> collect information in what environment which values are effective and
>>> >> which are not. Would you like to help us with that?
>>> >>
>>> >>
>>> >> --
>>> >> ,,,^..^,,,
>>> >>
>>>
>>
>>
>

--047d7b6731faaf417104ea558f2d--