couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jan Lehnardt (Commented) (JIRA)" <>
Subject [jira] [Commented] (COUCHDB-1342) Asynchronous file writes
Date Wed, 16 Nov 2011 20:49:51 GMT


Jan Lehnardt commented on COUCHDB-1342:

> I disagree fairly strongly on this point. couch_file is about as core of an API as it
gets. couch_file:flush/1 shouldn't even exist as far as I'm concerned. This isn't a mediocre
API that should be improved later, its a bad API that shouldn't go in at all. Its 100% foot-gun.

The new issue would be definitely a blocker for the next release. I don't see a problem with
this, but I'm happy to iterate the patch in JIRA as well.

> I don't need another patch to know that this one is complicated and could be less complicated.

Earlier versions of this patch did use gen_servers and they were neither less complicated
nor did they give the desired performance improvements, thus we went past them. They add a
lot of overhead for such a low level module. So yes, I think we need an alternative patch
to see if it were simpler and as useful.

> You're pointing at code that's about three abstractions away from couch_file's writer
loop. Suffice it to say, erlang:monitor/2 on a random process in the write path does little
to assuage my fears that we're entering dangerous territory relying on our own file writing

Not sure where there this is three abstractions away, but I gotta have to defer to the experts
Filipe and Damien here.
> Asynchronous file writes
> ------------------------
>                 Key: COUCHDB-1342
>                 URL:
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>            Reporter: Jan Lehnardt
>             Fix For: 1.3
>         Attachments: COUCHDB-1342.patch
> This change updates the file module so that it can do
> asynchronous writes. Basically it replies immediately
> to process asking to write something to the file, with
> the position where the chunks will be written to the
> file, while a dedicated child process keeps collecting
> chunks and write them to the file (and batching them
> when possible). After issuing a series of write request
> to the file module, the caller can call its 'flush'
> function which will block the caller until all the
> chunks it requested to write are effectively written
> to the file.
> This maximizes the IO subsystem, as for example, while
> the updater is traversing and modifying the btrees and
> doing CPU bound tasks, the writes are happening in
> parallel.
> Originally described at
> Github Commit:

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message