couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Joseph Davis (Commented) (JIRA)" <>
Subject [jira] [Commented] (COUCHDB-1342) Asynchronous file writes
Date Wed, 16 Nov 2011 16:48:01 GMT


Paul Joseph Davis commented on COUCHDB-1342:

This patch looks like its a mishmash of a couple patches that are applied slightly differently.
Not sure if that's a branch thing or what. But for instance, there's a lot of id_btree(Db)
-> Db#db.by_docid_btree type changes. There's also a switch between couch_file:sync(Filepath)
and couch_file:sync(Fd). I'm not sure if that's on purpose or not because we switched to Filepath
on purpose. I see it as being possible but I don't see a reference to it.

I'm really not a fan of exposing the inner workings of couch_file to every client as can be
seen by the massive number of flush calls that this creates. I find it quite likely that we'll
eventually end up missing one of these flush calls and causing a nasty bug because the error
would look like data hadn't been written. I'd want to see things rewritten so that the usage
of couch_file doesn't change. Easiest solution I see would be to automatically flush buffers
if a read request comes in for something that's not on disk.

I'm pretty sure we talked about this at one point, but can someone describe the differences
between this and Erlang's delayed_write mode for files?

couch_file becomes much more complicated with this patch. I'd prefer to see it simplified
if at all possible. I'm not entirely certain how it'd look but I might start by making a couch_file_reader
and couch_file_writer gen_servers instead of having bare processes in couch_file.

I have to say that this patch scares me a bit. If we make the switch to something like this
then we're opening up a whole new type of failure scenario where file "writes" appear to succeed
but then end up failing after the caller has moved on. While there are certainly cases where
I can see this being a fine tradeoff (view updaters, compactors, etc) it worries me a bit
for the main database file. The fact is that I can't (yet) reasonably consider all of the
possible new failure modes and convince myself that things are correct. I've already seen
some super weird reactions to things like running out of disk space, it seems like not knowing
about such an error until you've "written" a megabyte of data seems awkward.

> Asynchronous file writes
> ------------------------
>                 Key: COUCHDB-1342
>                 URL:
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>            Reporter: Jan Lehnardt
>             Fix For: 1.3
>         Attachments: COUCHDB-1342.patch
> This change updates the file module so that it can do
> asynchronous writes. Basically it replies immediately
> to process asking to write something to the file, with
> the position where the chunks will be written to the
> file, while a dedicated child process keeps collecting
> chunks and write them to the file (and batching them
> when possible). After issuing a series of write request
> to the file module, the caller can call its 'flush'
> function which will block the caller until all the
> chunks it requested to write are effectively written
> to the file.
> This maximizes the IO subsystem, as for example, while
> the updater is traversing and modifying the btrees and
> doing CPU bound tasks, the writes are happening in
> parallel.
> Originally described at
> Github Commit:

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message