incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boaz Citrin <bcit...@gmail.com>
Subject Re: Finding the right value for compaction configuration parameters
Date Sun, 03 Nov 2013 08:02:28 GMT
More numbers...

Same database, this time on a desktop, Windows Server 2008 R2, 2GB RAM,
Inter Xeon 3050 2*13 GHz.

checkpoint_after |doc_buffer_size | avg. compaction time (in minutes)
====================================================================
41943040            | 4194304             | 17:26
83886080            | 8388608             | 12:01
167772160          | 16777216          | 9:34
335544320          | 33554432          | 8:56
671088640          | 67108864          | 9:26 (*)
1342177280        | 134217728       | 10:20
2684354560        | 268435456       | 12:28

671088640          | 4194304            | 17:25
671088640          | 8388608            | 12:17
671088640          | 16777216          | 10:15
671088640          | 33554432          | 8:56
671088640          | 67108864          | 9:20 (*)
671088640          | 134217728        | 10:16
671088640          | 268435456        | 12:19





On Fri, Nov 1, 2013 at 5:44 PM, Boaz Citrin <bcitrin@gmail.com> wrote:

> OK some numbers when running compaction of a database of 1.5M docs on a
> Dell Latitude laptop with Windows 7 64b OS, 8G RAM, Intel Core i7-3720QM
> CPU (2 * 2.6 GHz).
>
> checkpoint_after |doc_buffer_size | avg. compaction time (in minutes)
> ======================================================
> 41943040            | 4194304             | 6:11
> 83886080            | 8388608             | 5:10
> 167772160          | 16777216          | 4:49
> 335544320          | 33554432          | 4:06
> 671088640          | 67108864          | 3:34 (*)
> 1342177280        | 134217728       | 4:16
> 2684354560        | 268435456       | 5:19
>
> 671088640          | 4194304            | 5:39
> 671088640          | 8388608            | 4:06
> 671088640          | 16777216          | 3:44
> 671088640          | 33554432          | 3:20
> 671088640          | 67108864          | 3:16 (*)
> 671088640          | 134217728        | 3:44
> 671088640          | 268435456        | 4:28
>
> As you can see in the starred lines, same parameters might give different
> results, but I think the direction is clear;
> In my situation buffer of 64m gives the fastest compaction, and we also
> see that as expected the checkpoint size affects the compaction time as
> well.
>
> Note that this is NOT the time that an average daily compaction takes as I
> ran these consecutively, i.e. the database was initially not fragmented.
>
>
>
> On Sat, Oct 26, 2013 at 5:25 PM, Alexander Shorin <kxepal@gmail.com>wrote:
>
>> Feel free to post here! I think others would be also interested in
>> your experience and could share (I hope so) their own too.
>> --
>> ,,,^..^,,,
>>
>>
>> On Sat, Oct 26, 2013 at 7:05 PM, Boaz Citrin <bcitrin@gmail.com> wrote:
>> > Hi Alexander,
>> >
>> > Yes, I have some numbers, do you want me to share here or somewhere
>> else?
>> >
>> > Best,
>> >
>> > Boaz
>> >
>> >
>> > On Mon, Oct 21, 2013 at 12:32 AM, Alexander Shorin <kxepal@gmail.com>
>> wrote:
>> >
>> >> Hi Boaz!
>> >>
>> >> On Fri, Oct 18, 2013 at 12:57 AM, Boaz Citrin <bcitrin@gmail.com>
>> wrote:
>> >> > Testing database compaction with various doc_buffer_size values I get
>> >> > completely different results.
>> >> > Documentation is fairly vague, so I wonder how to choose the right
>> value;
>> >> > What parameters affect this - HD buffer size, average doc size,
>> >> > database size, fragmentation, etc... ?!
>> >> >
>> >> > Same goes for view compaction and keyvalue_buffer_size.
>> >>
>> >> These parameters does affect on what they are named: they defines
>> >> buffer size for copying data from db/view file to the .compact one.
>> >> What information you think is missed?
>> >>
>> >> > (For me the the compaction with default values was many times slower
>> >> > than with the values that gave the faster compaction).
>> >>
>> >> All numbers have their cost: large buffers requires more memory while
>> >> they reduces I/O operations and vice versa. Much likely, that default
>> >> values wouldn't provide you high performance since they aimed to fit
>> >> everyone, but that's why you may tweak them for your needs (:
>> >>
>> >> I believe, that it's possible to revise them, but first need to
>> >> collect information in what environment which values are effective and
>> >> which are not. Would you like to help us with that?
>> >>
>> >>
>> >> --
>> >> ,,,^..^,,,
>> >>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message