couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Kocoloski (JIRA)" <>
Subject [jira] Commented: (COUCHDB-487) Pause write requests to allow compaction to complete
Date Thu, 27 Aug 2009 13:17:59 GMT


Adam Kocoloski commented on COUCHDB-487:

I don't think we want to be that aggressive about blocking incoming writes.  This algorithm
would cause a large DB to be unavailable for writes for several minutes, even if the write
load is not so large as to prevent compaction from finishing without blocking.  The reason
is that each compaction iteration works with a single MVCC snapshot; any updates to the DB
that occur during that iteration are compacted in the next pass.

I've seen several cases where compaction finishes in 3-4 passes, with the first one taking
several hours, the next one several minutes, the next one several seconds, and then it's done.

I think if we're going to block writes we should only do it when we have some expectation
that the non-blocking approach isn't working; e.g. we could track (update_seq - compact_update_seq)
and if that difference grows between iterations, block until compaction is finished.

Thanks for kicking off this ticket with some code, though!

> Pause write requests to allow compaction to complete
> ----------------------------------------------------
>                 Key: COUCHDB-487
>                 URL:
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>            Reporter: Robert Newson
>         Attachments: block_writes_after_end_of_compaction.patch
> Continuous writes will currently indefinitely prevent the compaction process from flipping
over to the .compact file.
> Here's a small patch that adds a new db flag called 'write_available' which becomes false
the first time compaction completes but fails to flip over because of concurrent writes. Subsequent
calls to update_docs then sleep for 1/2 a second and throw retry.
> I ran 'ab -p json -n 1000000 http://localhost:5984/db1' in one window. Without this patch,
I get a long sequence of;
> [info] [<0.64.0>] Compaction file still behind main file (update seq=1848. compact
update seq=1845). Retrying.
> and compaction never completes.
> With the patch, I get this;
> [info] [<0.369.0>] - - 'POST' /db1 201
> [info] [<0.65.0>] Compaction file still behind main file (update seq=2140. compact
update seq=2092). Retrying.
> [info] [<0.65.0>] Blocking writes to complete compaction.
> [info] [<0.65.0>] Compaction for db "db1" completed.
> [info] [<0.370.0>] - - 'POST' /db1 201

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message