couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Kocoloski (JIRA)" <>
Subject [jira] Commented: (COUCHDB-738) more efficient DB compaction (fewer seeks)
Date Tue, 04 May 2010 03:38:55 GMT


Adam Kocoloski commented on COUCHDB-738:

Ok, here's a first pass.  I ran the usual script which loads 1000 'simple' and 1000
'unsimple' docs into a DB and computes the MegaView and SimpleView.  I modified the script
to update each document 100 times instead of only once.  Here are the numbers for trunk:

DB size: 1.1G pre-compaction, 12MB post

Elapsed time: 7.20 (s)
Elapsed time: 5.65 (s)
Elapsed time: 5.13 (s)

Elapsed time: 0.75 (s)
Elapsed time: 1.04 (s)
Elapsed time: 0.74 (s)

Indexing times after compaction
Elapsed time: 4.91 (s)
Elapsed time: 0.74 (s)

Note that only the first measurement included a full load_docs step.  For the latter measurements
I just deleted and rebuilt the index files (updating the docs 100 times took several minutes,
and I didn't feel like waiting).  Here are the equivalent numbers with the patch:

DB size: 1.5G pre-compaction, 17MB post

Elapsed time: 9.03 (s)
Elapsed time: 8.22 (s)
Elapsed time: 5.92 (s)

Elapsed time: 0.99 (s)
Elapsed time: 0.94 (s)
Elapsed time: 0.91 (s)

After compaction
Elapsed time: 5.81 (s)
Elapsed time: 0.90 (s)

There seems to be quite a bit of variance in these numbers.  I usually take the minimum when
benchmarking on my laptop, although due to my skipping the load_docs step the minimum is probably
a little too good (the seq_tree is fully cached).  Anyway, if I stick with that methodology
I'd say that for DBs with 100 edits/document we see

MegaView: 15-18% slower with patch 
SimpleView: 22-23% slower with patch

The database is 35-40% larger, too.

I might try loading a 500 edits / document DB tonight.  I may also look at eprof to see if
there are simple optimizations for this use case.  Is the number of edit branches significant?
 So far I haven't been introducting any conflicts in the test.

> more efficient DB compaction (fewer seeks)
> ------------------------------------------
>                 Key: COUCHDB-738
>                 URL:
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>    Affects Versions: 0.9.2, 0.10.1, 0.11
>            Reporter: Adam Kocoloski
>            Assignee: Adam Kocoloski
>             Fix For: 1.1
>         Attachments: 738-efficient-compaction-v1.patch
> CouchDB's database compaction algorithm walks the by_seq btree, then does a lookup in
the by_id btree for every document in the database.  It does this because the #full_doc_info{}
record with the full revision tree is only stored in the by_id tree.  I'm proposing instead
to store duplicate copies of #full_doc_info{} in both trees, and to have the compactor use
the by_seq tree exclusively.  The net effect is significantly fewer calls to pread(), and
an compaction IO pattern where reads tend to be clustered close to each other in the file.
> If the by_id tree is fully cached, or if the id tree nodes are located near the seq tree
nodes, the performance improvement is small but noticeable (~10% in some simple tests).  On
the other hand, in the worst-case scenario of randomly-generated docids and a database much
larger than main memory the improvement is huge.  Joe Williams did some simple benchmarks
with a 50k document, 600 MB database on a 256MB VPS.  The compaction time for that DB dropped
from 15m to 2m20s, so more than 6x faster.
> Storing the #full_doc_info{} in the seq tree also allows for some similar optimizations
in the replicator.
> This patch might have downsides when documents have a large number of edits.  These include
an increase in the size of the database and slower view indexing.  I expect both to be small
> The patch can be applied directly to trunk@934272.  Existing DBs are still readable,
new updates will be written in the new format, and databases can be fully upgraded by compacting.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message