couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Joseph Davis (Commented) (JIRA)" <>
Subject [jira] [Commented] (COUCHDB-1342) Asynchronous file writes
Date Thu, 17 Nov 2011 01:41:53 GMT


Paul Joseph Davis commented on COUCHDB-1342:


That's an awful lot of disappointment packed into a single comment. First, resorting to an
ad hominem attack to insinuate that I'm not intelligent enough to work on databases is quite
disconcerting. Secondly, its an egregious fallacy to suggest that because a patch appears
to be technically correct that it should be committed. Thirdly, declaring what is and isn't
a valid reason to hold up a patch is not how the ASF works.

And now back to the regularly scheduled technical discussion.

First, couch_file:flush/1. Unless I'm missing something extremely subtle here, it's existence
is so that clients can read their own writes. Yet the couch_file gen_server has all the knowledge
it needs to know if it has to flush to service a write call. If the requested read position
is between #file.eof and #file.eof + #file.queued_write_bytes, then it can call flush and
move on with its life. Not only does this mean that clients don't have remember to call flush,
but it removes unnecessary message passing that every unconditional call to flush would generate.

Second, this is doubling the number of file descriptors required for anything that isn't a
database. On the first production machine I checked that's an increase of 75% from 40K to
70K file descriptors. That's a fairly serious change that ought to be discussed. At the very
least it ought to be mentioned somewhere so ops teams know to expect it.

Third, this is spawning long lived processes that aren't looping on exported functions. After
two code upgrades this would crash every couch_file in the VM simultaneously.

Fourth, as I've mentioned numerous times before, the proper way to synchronously start a process
that might fail to initialize is to use proc_lib:start_link and proc_lib:init_ack.

Fifth, has anyone considered using a write buffer outside of the couch_file API that would
allow clients more precise control. For instance, thinking briefly on the view updater, you
could buffer writes for a single add_remove call. This also leads to the possibility that
mostly read views aren't needlessly holding open a writer fd  for no reason.

> Asynchronous file writes
> ------------------------
>                 Key: COUCHDB-1342
>                 URL:
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>            Reporter: Jan Lehnardt
>             Fix For: 1.3
>         Attachments: COUCHDB-1342.patch
> This change updates the file module so that it can do
> asynchronous writes. Basically it replies immediately
> to process asking to write something to the file, with
> the position where the chunks will be written to the
> file, while a dedicated child process keeps collecting
> chunks and write them to the file (and batching them
> when possible). After issuing a series of write request
> to the file module, the caller can call its 'flush'
> function which will block the caller until all the
> chunks it requested to write are effectively written
> to the file.
> This maximizes the IO subsystem, as for example, while
> the updater is traversing and modifying the btrees and
> doing CPU bound tasks, the writes are happening in
> parallel.
> Originally described at
> Github Commit:

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message