couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Stockton" <>
Subject RE: CouchDB 1.2 Replicator Changes
Date Fri, 27 Apr 2012 18:53:30 GMT

Sorry for the unfinished previous email.

> Chris, I am working on this to support users with tens of thousands of
> continuous replications at Iris Couch.

> Are you able to run Node.js software in production, by any chance? The
> solution we are working on is a Node.js monitor which triggers
> replications as needed to get the effect of many continuous
> replications running, but without the resource usage. If Node.js is on
> the same server as CouchDB, it can defer to the kernel (inotify) and
> be very efficient keeping up with system-wide updates.

I think we would be able to run node js if a solution was built that
could fit our needs. The thing is every time I come back to our
replication issues I find other people solving it through some kind of
custom solution, using application knowledge or other means of polling
etc to flag and monitor databases.

We do indeed already have a daemon that runs making sure all databases
running have continuous replication running, as well as doing some
automatic failover and recovery stuff. Would it be possible to maybe try
to roll off inactive databases? Certainly but we aim for "NO" data loss,
ever. If our Master server for any reason ever dies completely, we
trigger a automatic failover and customers may notice a small delay in a
single request at most, but they will never have a service interruption
and margin for data loss is very small. This is thanks to continuous
replication and the overall design of CouchDB which we love very much.
Now a database can be modified at any point and time, we don't have a
list of "active" sessions or anything of the nature, a database can be
written to anytime. We don't track the states of each individual write
or mark any kind of flags.

By the time we implemented some custom logic to roll off replicating
inactive databases, and start them backup when written too we would have
a very complex and error prone model, that wouldn't give us the
durability and reliability we have come to know with the continuous
all-dbs model. Now maybe with some more thought and careful planning
this isn't entirely true, but my point here is that the best, most
efficient place for this kind of functionality is inside CouchDB.

I saw a voting for server wide _changes feed, that would be amazing and
I hope that picks up some traction, I would gladly donate my time to
such a feature but unfortunately I don't know erlang, and the few times
I have looked into couchdb source to try to figure things out I am lost.
The syntax is very foreign from the languages I am use to.

TLDR; We would run a solution to our problem if we could find one, or
hand roll our own in our current replication monitor, but I don't think
anything could be as good as a built-in feature (Global changes feed /
server wide replication).


View raw message