couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: small worker_batch_size
Date Tue, 10 Dec 2013 18:06:28 GMT
On Dec 10, 2013, at 12:30 PM, Jan Lehnardt <jan@apache.org> wrote:

> Heya Adam,
> 
> I found this on IRC:
> 
>  <kocolosk> [Thu 23:43] dch: yeah i recommend a small worker_batch_size for docs
with attachments for other reasons
> 
> Could you elaborate on the other reasons? :)
> 
> Best
> Jan
> -- 

Sure:

* The entire batch gets blocked on the transfer of a single document with attachments, so
in the limiting case you end up with only worker_processes documents being transferred in
parallel.

* A batch might contain several documents with large attachments; these will be processed
in series.

With those in mind we've found that we can get the best attachment replication throughput
out of our Cloudant clusters when we decrease the batch size (to limit the queue depth) and
up the worker_processes count a bit (to improve parallelism).  I can't promise that those
suggestions are immediately applicable to all Apache CouchDB servers, but the facts motivating
the adjustment do hold true in both codebases.

Adam
Mime
View raw message