couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Harm <cont...@aharm.de>
Subject Re: Max number of continuous replications supported by Couch?
Date Tue, 29 Mar 2016 09:47:21 GMT
Good to know. I implemented now a replication queue that manages pending replications and active
replication processes.

Alexander

> On 28. Mar. 2016, at 15:43, Robert Samuel Newson <rnewson@apache.org> wrote:
> 
> 
> This is something we're working on. At present, the replication manager will try to run
all replications at once. We plan to introduce a smarter job scheduler so that you can configure
'max_replications_running' independently of other concerns.
> 
> For now, you could use non-continuous replications and listen to /_db_updates on your
sources to know when to trigger those replications. This is, roughly speaking, what the replication
manager will do in a future release.
> 
> B.
> 
> 
>> On 28 Mar 2016, at 09:01, Alexander Harm <contact@aharm.de> wrote:
>> 
>> Hello all,
>> 
>> I’ve been playing with CouchDB over the last weeks and am running into serious
problems. My idea was, to have a central database and roughly 1.000 project dbs that stem
from a filtered local one-way replication from the central db using _replicator db. However,
triggering 1.000 replications grinds CouchDB to a standstill. I read that for each replication
Couch spawns a process so that approach might not be feasible after all.
>> I changed my setup now into a node process listening to the changes feed of the central
database and then manually triggering a one-shot replication to the affected project dbs.
That seems to work for now.
>> 
>> However, now the clients should listen to the changes feed of their relevant project
dbs and again CouchDB is thrashing and crashing all over the place. I increased "max_dbs_open"
to 1.500 and added "+P 10.000” to the Erlang startup parameters. ULIMIT on OS X is set to
unlimited.
>> 
>> My central database only has 1.000 small documents so I’m a bit scared of what
will happen if it grows to millions of documents and hundreds of clients listening to the
changes feeds.
>> 
>> Are there any recommendations on (nr_of_dbs, nr_of_clients, nr_of_replications) =>
{couch_settings, erlang_settings, system_settings}?
>> Or should I not even attempt to host Couch myself (I would really like to) and directly
use Cloudant?
>> 
>> I’m using CouchDB 1.6.1 (installed via homebrew) on OS X 10.11.4
>> 
>> Any help appreciated.
>> 
>> Regards, Alexander
> 


Mime
View raw message