incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randall Leeds <>
Subject Re: dbcopy - A faster alternative to replication
Date Sat, 22 Jan 2011 00:22:01 GMT
On Fri, Jan 21, 2011 at 15:58, Chris Stockton <> wrote:
> Hello,
> After some discussion about user migration and issues with 1GIG+ sized
> databases, it seems we will not be able to use replication for copying
> databases. We have seen the speed offered by rsync/cp in comparison to
> replication in couchdb. Our thoughts currently lie in a external
> handler to make this a bit easier, rsync / ssh user permissions across
> our server draws away from some of our security methodologies but it
> is workable.
> My question is could(or would rather) a efficient database copy ever
> be implemented in couch similar to the way replication API works? One
> that optimized both local server and remote server database copying.
> If no future plans exist what would be the possible implementation
> details, roadmap/business rules holding such a feature back? Does
> anyone else have any sort of need or desire for faster, built in
> database copying?
> Kind Regards,
> -Chris

Hey, Chris.

First, you might check out Filipe's work on a new replicator[1]. He
reports some significant speedups in some cases[2].

Understand that while scp/rsync works for copying a database it does
not work for incremental updates. You can rsync until you compact, but
then everything will have to be copied from scratch. The replication
protocol does not suffer from this limitation.

A more interesting project, in my opinion, would be to experiment with
ways to replace the existing replication with a more efficient
transport. BigCouch[3] might have some code you could take for this
and I'm working on ways to make it easier to develop new Erlang
modules for CouchDB[4] so you could plug in a custom replicator.

There's definitely work to be done here if you want to get involved. I
think many people would like a faster replicator. I certainly would!


View raw message