couchdb-replication mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <adam.kocolo...@gmail.com>
Subject Re: Checkpointing on read only databases
Date Sun, 13 Apr 2014 17:50:01 GMT
Yeah, this is a subtle little thing. The main reason we checkpoint on both source and target
and compare is to cover the case where the source database is deleted and recreated in between
replication attempts. If that were to happen and the replicator just resumes blindly from
the checkpoint sequence stored on the target then the replication could permanently miss some
documents written to the new source.

I'd love to have a robust solution for incremental replication of read-only databases. To
first order a UUID on the source database that was fixed at create time could do the trick,
but we'll run into trouble with file-based backup and restores. If a database file is restored
to a point before the latest replication checkpoint we'd again be in a position of potentially
permanently missing updates.

Calvin's suggestion of storing e.g. {seq, id, rev} instead of simply seq as the checkpoint
information would dramatically reduce the likelihood of that type of permanent skip in the
replication, but it's only a probabilistic answer.

Adam

On Apr 13, 2014, at 1:31 PM, Calvin Metcalf <calvin.metcalf@gmail.com> wrote:

> Though currently we have the opposite problem right if we delete the target
> db? (this on me brain storming)
> 
> Could we store last rev in addition to last seq?
> On Apr 13, 2014 1:15 PM, "Dale Harvey" <dale@arandomurl.com> wrote:
> 
>> If the src database was to be wiped, when we restarted replication nothing
>> would happen until the source database caught up to the previously written
>> checkpoint
>> 
>>   create A, write 5 documents
>>   replicate 5 documents A -> B, write checkpoint 5 on B
>>   destroy A
>>   write 4 documents
>>   replicate A -> B, pick up checkpoint from B and to ?since=5
>>   .. no documents written
>> 
>> 
>> https://github.com/pouchdb/pouchdb/blob/master/tests/test.replication.js#L771is
>> our test that covers it
>> 
>> 
>> On 13 April 2014 18:02, Calvin Metcalf <calvin.metcalf@gmail.com> wrote:
>> 
>>> If we were to unilaterally switch to checkpoint on target what would
>>> happen, replication in progress would loose their place?
>>> On Apr 13, 2014 11:21 AM, "Dale Harvey" <dale@arandomurl.com> wrote:
>>> 
>>>> So with checkpointing we write the checkpoint to both A and B and
>> verify
>>>> they match before using the checkpoint
>>>> 
>>>> What happens if the src of the replication is read only?
>>>> 
>>>> As far as I can tell couch will just checkout a checkpoint_commit_error
>>> and
>>>> carry on from the start, The only improvement I can think of is the
>> user
>>>> specifies they know the src is read only and to only use the target
>>>> checkpoint, we can 'possibly' make that happen automatically if the src
>>>> specifically fails the write due to permissions.
>>>> 
>>> 
>> 


Mime
View raw message