incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Goodall <matt.good...@gmail.com>
Subject Re: Incremental replication over unreliable link -- how granular is replication restart
Date Thu, 14 May 2009 16:00:42 GMT
2009/5/14 Adam Kocoloski <kocolosk@apache.org>:
> Hi Matt,
>
> On May 14, 2009, at 10:36 AM, Matt Goodall wrote:
>
>> 2009/5/14 Adam Kocoloski <kocolosk@apache.org>:
>>>
>>> Hi Ben, welcome!  At the moment, CouchDB does not have any capacity for
>>> intra-document replication checkpointing.  And you're right, in the
>>> specific
>>> situation you describe Couch would have a difficult time making any
>>> replication progress.
>>>
>>> Given that replication over slow, unreliable links is absolutely a
>>> CouchDB
>>> design goal, I think we might eventually conjure up some more magic to
>>> make
>>> some sort of intra-document (or at least intra-attachment) checkpointing
>>> possible.  I think it will be post-1.0, though.  Best,
>>>
>>> Adam
>>>
>>> On May 14, 2009, at 7:12 AM, Ben Cohen wrote:
>>>
>>>> Hi all --
>>>>
>>>> This is my first message to the list.  I've been watching it for a
>>>> little
>>>> while now and so far everything I read about the design of couchdb I
>>>> like a
>>>> lot!  Thanks so much for all the cool work!
>>>>
>>>> One of the uses I'm planning for couchdb involves replicating a database
>>>> across a slow, unreliable link which will never become anything other
>>>> than
>>>> slow and unreliable.  I understand the replication is incremental and
>>>> designed to 'pick up where it left off' in the case of replication
>>>> interruption.  From the technical overview on the website:
>>>>
>>>>> The replication process is incremental. At the database level,
>>>>> replication only examines documents updated since the last replication.
>>>>> Then
>>>>> for each updated document, only fields and blobs that have changed are
>>>>> replicated across the network. If replication fails at any step, due
to
>>>>> network problems or crash for example, the next replication restarts
at
>>>>> the
>>>>> same document where it left off.
>>
>> Is this actually accurate? It suggests that documents are replicated
>> one-by-one and that replication can be interrupted at any point and
>> will continue from wherever it got to before the interruption.
>
> Yes, there are some inaccuracies in that paragraph.  We do save checkpoints,
> but not per-document.  We also transfer the whole document, not just changed
> fields.  In some respects the Overview is really part Roadmap.  We've taken
> some flak for this before, perhaps it's time to revisit that page.
>
>> Firstly, I believe the whole replication has to complete before any
>> updates are visible in the target database. If I restart the server in
>> charge of replication and then restart the replication it always seems
>> to start from the beginning. i.e. the Futon's "Processed source update
>> #xxx" status starts from 0 (when replicating an empty database).
>
> The exact behavior has changed as CouchDB has evolved.  Are you running 0.9
> or higher?  In that case Couch should be saving checkpoints for every ~10MB
> of document data that comes across the wire.  If it fails after a
> checkpoint, the next replication should not be starting from 0.  If it is
> restarting, I'd consider that a bug.

When I tried things before writing my mail I was using two couchdb
servers running from relatively recent versions of trunk. So 0.9 and a
bit ;-).

I didn't know about the ~10MB. I don't know if I reached that
threshold which may be why it seemed to be started over each time.
I'll try to retest with a lower threshold and more debugging to see
what's really happening. Any help on where that hard-coded 10MB value
is would be very helpful!

>
> Others have commented that the 10MB threshold really needs to be
> configurable.  E.g., set it to zero and you get per-document checkpoints,
> but your throughput will drop and the final DB size on the target will grow.
>  Super easy to do, but no one's gotten around to it.

Presumably the threshold all depends on the quality of the network
connection between the two endpoints, although having the default
configurable is probably a good thing anyway.

>
>> Secondly, if the network connection fails in the middle of replication
>> (closing an ssh tunnel is a good way to test this ;-)) then it seems
>> to retry a few (10) times before the replicator process terminates. If
>> the network connection becomes available again (restart the ssh
>> tunnel) the replicator doesn't seem to notice. Also, I just noticed
>> that Futon still lists the replication on its status page.
>
> That's correct, the replicator does try to ignore transient failures.

Hmm, it seemed to fail on transient failures here. After killing and
restarting my ssh tunnel I left the replication a while and it never
seemed to continue, and the only way to clear it from the status list
was to restart the couchdb server. I'll check again though.

>
>> If I'm correct, and I really hope I'm missing something, then
>> couchdb's replication is probably not currently suitable for
>> replicating anything but very small database differences over an
>> unstable connection. Does anyone have any real experience in this sort
>> of scenario?
>
> Personally, I do not.  I think the conclusion is a bit pessimistic, though.

Sorry, wasn't meaning to be pessimistic. Just trying to report
honestly what I was seeing so it could be improved where possible.

>  Adding a configurable checkpoint threshold should make it possible to
> (slowly) replicate very large DB differences.  Ben's original point about
> the inability to replicate very large documents still stands, though.  I've
> opened a ticket to remind us about adding that feature in the future.
>  Cheers,
>
> Adam
>
>
>

Mime
View raw message