couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Stockton" <>
Subject CouchDB 1.2 Replicator Changes
Date Wed, 25 Apr 2012 18:53:51 GMT

I was very excited when I was reading the replicator changes in the
release notes[1], specifically because I saw "Number of worker
processes" and thought that maybe it is now pooled. Although I am very
glad to see the improvements to the replicator and very much appreciate
the work that has been done to it; I am a bit confused after reading the
more detailed paramters for the new replicator[2]. It seems that the
configuration options and worker processes are for a specific database,
with some decently high defaults, such as 20 "http_connections". From
what I gather reading this is per database, or is it per server?

I have sent emails in the past to this list how I would love to see a
server wide replicator, something that created a configurable pool of
connections for server relationships. For us, we scale with many
databases instead of having one giant database. The problem we have
faced is as we reached only 2K databases some configuration tweaks had
to be made to allow replication to run from our Master -> Failover ->
Backup machine, as we got up to 5000 we were forced to take our Backup
machine out of the picture due to putting around 10K TCP connection
requirement to our Fail over machine. It was simply to much strain even
for very large enterprise database servers.

So my question here is does the new replicator pool an entire server,
solving our growth problem with MANY databases, or does it simply add
additional strain with more workers (from 5000 tcp connections to 100k)?
If it does indeed add additional workers instead of lower them, if I was
to lower the defaults to 1 connection per database, is the new
replicator designed in such a way that it will still offer at least
comparable performance to the 1.1 replicator, or could I possibly incur
a penalty because the new architecture is designed and expected to have
a modest pool size?

Kind Regards,



View raw message