couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: CouchDB Filtered Replication - Group of Doc's Don't appear to be batched
Date Sun, 02 Aug 2015 00:25:28 GMT
Hi Bob,

The replicator will dynamically choose _bulk_docs batch sizes based on the number of documents
that are ready to be transmitted to the source. It’s possible to set an upper bound on the
size of the batch, but at this time it’s not possible to set a lower bound.

It sounds like what’s happening here is that the replicator is faster than the filter function,
and that it’s constantly waiting for the next document to pass the filter. One sanity check
you might try is to request the filtered _changes feed directly and see what kind of throughput
you get. The filtered _changes feed sets the upper bound on the replication throughput you
can achieve with that filter. If the _changes feed is fast but the replication is slow, the
next thing you should try to do is minimize the replication-related resource consumption on
the source — e.g., is the replication mediated by the server hosting the source database,
and if so do you have an opportunity to mediate the replication on a different server? Cheers,

Adam

> On Jul 24, 2015, at 10:50 PM, Bob Hathaway <bob_hathaway@ymail.com> wrote:
> 
> Our couchdb replication runs fast without filters to a couch instance.  But from that
> couch instance to another a filter is doing a simple check and sync'ing about 10x slower.
> Looking at couch debug logging, the bulk_docs Content-length is 10x smaller  with the
filter.
> With filter the _bulk_docs rate per minute appears identical to the number of docs sync'd.
> Without the filter, the _bulk_docs rate per minute is 10x less than docs sync'd.
> 
> It would appear the couch replication protocol groups 10 docs in a _bulk_docs POST without
a replication filter
> but  with the filter there is no grouping and bulk_docs appears to only contain a single
doc.
> 
> Does couchdb filter replication not group docs and send 1 doc in each _bulk_docs call
to the target host?
> 
> Is there some configuration which would allow the filter replication to group docs to
speed up replication?
> 
> -- 
> Robert Hathaway
> President & Chief Software Architect
> SOA Object Systems, LLC
> office:  201-408-5828
> cell:    201-390-7602
> email: rjhsoa@gmail.com
> 


Mime
View raw message