Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of matt.goodall@gmail.com
 designates 209.85.219.168 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=AXu+DhfJSmYqGr2ity95mzVz0gxf5bmNQb71p5dQMJS3ds8VtbkEO96RYf1rIqN91O
         daelej7/NONsYp3Ked0WppfoYu9lkacquspBA9r3+iK+vqsvORlmoUi5coN0VVu6We00
         sst3GleRSadCRm/DmmStEUTfbdY1cp1BtqC9I=
MIME-Version: 1.0
In-Reply-To: <55A25503-03E5-47E7-A92C-1DDDC0CF9BA1@apache.org>
References: <A192849F-B7EE-48A2-B010-D63D171FC034@gmail.com>
	 <55A25503-03E5-47E7-A92C-1DDDC0CF9BA1@apache.org>
Date: Thu, 14 May 2009 15:36:08 +0100
Message-ID: <214c385b0905140736v39e0a752x606e56b532786f91@mail.gmail.com>
Subject: Re: Incremental replication over unreliable link -- how granular is
	replication restart
From: Matt Goodall <matt.goodall@gmail.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

2009/5/14 Adam Kocoloski <kocolosk@apache.org>:
> Hi Ben, welcome! =C2=A0At the moment, CouchDB does not have any capacity =
for
> intra-document replication checkpointing. =C2=A0And you're right, in the =
specific
> situation you describe Couch would have a difficult time making any
> replication progress.
>
> Given that replication over slow, unreliable links is absolutely a CouchD=
B
> design goal, I think we might eventually conjure up some more magic to ma=
ke
> some sort of intra-document (or at least intra-attachment) checkpointing
> possible. =C2=A0I think it will be post-1.0, though. =C2=A0Best,
>
> Adam
>
> On May 14, 2009, at 7:12 AM, Ben Cohen wrote:
>
>> Hi all --
>>
>> This is my first message to the list. =C2=A0I've been watching it for a =
little
>> while now and so far everything I read about the design of couchdb I lik=
e a
>> lot! =C2=A0Thanks so much for all the cool work!
>>
>> One of the uses I'm planning for couchdb involves replicating a database
>> across a slow, unreliable link which will never become anything other th=
an
>> slow and unreliable. =C2=A0I understand the replication is incremental a=
nd
>> designed to 'pick up where it left off' in the case of replication
>> interruption. =C2=A0From the technical overview on the website:
>>
>>> The replication process is incremental. At the database level,
>>> replication only examines documents updated since the last replication.=
 Then
>>> for each updated document, only fields and blobs that have changed are
>>> replicated across the network. If replication fails at any step, due to
>>> network problems or crash for example, the next replication restarts at=
 the
>>> same document where it left off.

Is this actually accurate? It suggests that documents are replicated
one-by-one and that replication can be interrupted at any point and
will continue from wherever it got to before the interruption.

Firstly, I believe the whole replication has to complete before any
updates are visible in the target database. If I restart the server in
charge of replication and then restart the replication it always seems
to start from the beginning. i.e. the Futon's "Processed source update
#xxx" status starts from 0 (when replicating an empty database).

Secondly, if the network connection fails in the middle of replication
(closing an ssh tunnel is a good way to test this ;-)) then it seems
to retry a few (10) times before the replicator process terminates. If
the network connection becomes available again (restart the ssh
tunnel) the replicator doesn't seem to notice. Also, I just noticed
that Futon still lists the replication on its status page.

If I'm correct, and I really hope I'm missing something, then
couchdb's replication is probably not currently suitable for
replicating anything but very small database differences over an
unstable connection. Does anyone have any real experience in this sort
of scenario?

- Matt

>>>
>> I've got a question about this process. =C2=A0Say you have a document to=
 be
>> replicated with a 1 megabyte attachment. =C2=A0A replication process sta=
rts, half
>> the doc is transferred successfully and then the connection dies. =C2=A0=
Assuming
>> no changes to the source doc, when the replication restarts will the
>> transfer start from the beginning of the document or will it pick up
>> somewhere within the doc?
>>
>> For my use case I have a slow link that will periodically come online fo=
r
>> a certain fixed amount of time and initiate a replication. =C2=A0If the
>> replication isn't incremental 'within' a single document, then a documen=
t in
>> the database above a certain size will for me, never make it across and =
I
>> would imagine cause the replication to never make forward progress ...
>>
>> Does couchdb's replication magic avoid the issue for me and eventually
>> transfer the document across my link?
>>
>> Thanks much,
>> Ben Cohen
>
>