couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <>
Subject Re: Incremental replication over unreliable link -- how granular is replication restart
Date Thu, 14 May 2009 13:44:40 GMT
Hi Ben, welcome!  At the moment, CouchDB does not have any capacity  
for intra-document replication checkpointing.  And you're right, in  
the specific situation you describe Couch would have a difficult time  
making any replication progress.

Given that replication over slow, unreliable links is absolutely a  
CouchDB design goal, I think we might eventually conjure up some more  
magic to make some sort of intra-document (or at least intra- 
attachment) checkpointing possible.  I think it will be post-1.0,  
though.  Best,


On May 14, 2009, at 7:12 AM, Ben Cohen wrote:

> Hi all --
> This is my first message to the list.  I've been watching it for a  
> little while now and so far everything I read about the design of  
> couchdb I like a lot!  Thanks so much for all the cool work!
> One of the uses I'm planning for couchdb involves replicating a  
> database across a slow, unreliable link which will never become  
> anything other than slow and unreliable.  I understand the  
> replication is incremental and designed to 'pick up where it left  
> off' in the case of replication interruption.  From the technical  
> overview on the website:
>> The replication process is incremental. At the database level,  
>> replication only examines documents updated since the last  
>> replication. Then for each updated document, only fields and blobs  
>> that have changed are replicated across the network. If replication  
>> fails at any step, due to network problems or crash for example,  
>> the next replication restarts at the same document where it left off.
> I've got a question about this process.  Say you have a document to  
> be replicated with a 1 megabyte attachment.  A replication process  
> starts, half the doc is transferred successfully and then the  
> connection dies.  Assuming no changes to the source doc, when the  
> replication restarts will the transfer start from the beginning of  
> the document or will it pick up somewhere within the doc?
> For my use case I have a slow link that will periodically come  
> online for a certain fixed amount of time and initiate a  
> replication.  If the replication isn't incremental 'within' a single  
> document, then a document in the database above a certain size will  
> for me, never make it across and I would imagine cause the  
> replication to never make forward progress ...
> Does couchdb's replication magic avoid the issue for me and  
> eventually transfer the document across my link?
> Thanks much,
> Ben Cohen

View raw message