couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boaz Citrin <bcit...@gmail.com>
Subject Re: Finding the right value for compaction configuration parameters
Date Fri, 01 Nov 2013 15:44:19 GMT
OK some numbers when running compaction of a database of 1.5M docs on a
Dell Latitude laptop with Windows 7 64b OS, 8G RAM, Intel Core i7-3720QM
CPU (2 * 2.6 GHz).

checkpoint_after |doc_buffer_size | avg. compaction time (in minutes)
======================================================
41943040            | 4194304             | 6:11
83886080            | 8388608             | 5:10
167772160          | 16777216          | 4:49
335544320          | 33554432          | 4:06
671088640          | 67108864          | 3:34 (*)
1342177280        | 134217728       | 4:16
2684354560        | 268435456       | 5:19

671088640          | 4194304            | 5:39
671088640          | 8388608            | 4:06
671088640          | 16777216          | 3:44
671088640          | 33554432          | 3:20
671088640          | 67108864          | 3:16 (*)
671088640          | 134217728        | 3:44
671088640          | 268435456        | 4:28

As you can see in the starred lines, same parameters might give different
results, but I think the direction is clear;
In my situation buffer of 64m gives the fastest compaction, and we also see
that as expected the checkpoint size affects the compaction time as well.

Note that this is NOT the time that an average daily compaction takes as I
ran these consecutively, i.e. the database was initially not fragmented.



On Sat, Oct 26, 2013 at 5:25 PM, Alexander Shorin <kxepal@gmail.com> wrote:

> Feel free to post here! I think others would be also interested in
> your experience and could share (I hope so) their own too.
> --
> ,,,^..^,,,
>
>
> On Sat, Oct 26, 2013 at 7:05 PM, Boaz Citrin <bcitrin@gmail.com> wrote:
> > Hi Alexander,
> >
> > Yes, I have some numbers, do you want me to share here or somewhere else?
> >
> > Best,
> >
> > Boaz
> >
> >
> > On Mon, Oct 21, 2013 at 12:32 AM, Alexander Shorin <kxepal@gmail.com>
> wrote:
> >
> >> Hi Boaz!
> >>
> >> On Fri, Oct 18, 2013 at 12:57 AM, Boaz Citrin <bcitrin@gmail.com>
> wrote:
> >> > Testing database compaction with various doc_buffer_size values I get
> >> > completely different results.
> >> > Documentation is fairly vague, so I wonder how to choose the right
> value;
> >> > What parameters affect this - HD buffer size, average doc size,
> >> > database size, fragmentation, etc... ?!
> >> >
> >> > Same goes for view compaction and keyvalue_buffer_size.
> >>
> >> These parameters does affect on what they are named: they defines
> >> buffer size for copying data from db/view file to the .compact one.
> >> What information you think is missed?
> >>
> >> > (For me the the compaction with default values was many times slower
> >> > than with the values that gave the faster compaction).
> >>
> >> All numbers have their cost: large buffers requires more memory while
> >> they reduces I/O operations and vice versa. Much likely, that default
> >> values wouldn't provide you high performance since they aimed to fit
> >> everyone, but that's why you may tweak them for your needs (:
> >>
> >> I believe, that it's possible to revise them, but first need to
> >> collect information in what environment which values are effective and
> >> which are not. Would you like to help us with that?
> >>
> >>
> >> --
> >> ,,,^..^,,,
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message