Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of adam.kocoloski@gmail.com
 designates 209.85.132.251 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:mime-version:subject:date:references
         :x-mailer;
        b=aCXkTMOzBrQj5iVi63KaiLuN69nsOiX/TRidtppRFathdhclBUwwfOFQhruHD9lwqn
         Ql9aPDyUuCJ3HCy8MLxyy7s4Wgr5TUymTg3ni8Z8MhMDZVrN+9Ow/hoVBK32VxUxq6Iy
         9rSLxjE86ROLMxx+IbTeT3HlHdgFJWwQepZjg=
Message-Id: <C3974490-5AE9-456F-8FCD-7924B66488E3@gmail.com>
From: Adam Kocoloski <adam.kocoloski@gmail.com>
To: dev@couchdb.apache.org
In-Reply-To: <C613B84C-17EC-49CD-B440-534D740DCB32@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: handling simultaneous identical replications
Date: Thu, 5 Mar 2009 08:00:52 -0500
References: <3299B0C3-3038-4B6F-8B2F-B40885A7E8F4@gmail.com>
 <e2111bbb0903042231yc1ec310ra5787a580b000e8a@mail.gmail.com>
 <C613B84C-17EC-49CD-B440-534D740DCB32@apache.org>

On Mar 5, 2009, at 7:24 AM, Jan Lehnardt wrote:

>
> On 5 Mar 2009, at 07:31, Paul Davis wrote:
>
>> On Wed, Mar 4, 2009 at 8:34 PM, Adam Kocoloski <adam.kocoloski@gmail.com 
>> > wrote:
>>> Hi folks, we've been running into a problem where multiple  
>>> replications with
>>> the same source and target are running simultaneously.  This  
>>> introduces
>>> quite a lot of unnecessary network traffic and causes real  
>>> problems with
>>> update collisions on the local replication history documents.  If  
>>> replicator
>>> A updates the source doc and replicator B updates the target doc,  
>>> subsequent
>>> replications will decide that a full replication is necessary.
>>>
>>> I have some ideas about how to ensure only one is running at a  
>>> time (more on
>>> that in a separate mail), but I'd like some feedback on how to  
>>> handle the
>>> second..Nth request.  Let's call the initial POST to _replicate  
>>> "A" and the
>>> second POST "B":
>>>
>>> Option 1 -- Respond to B with the results from A
>>> This option works fine if the source is remote.  However, if the  
>>> source is
>>> local, the replication started by A will be missing updates to the  
>>> source DB
>>> that occurred between A and B.  B may be surprised by that result.
>>>
>>> Option 2 -- Grab an updated DB and continue the replication
>>> This option will include updates to the source that occurred  
>>> between A and B
>>> in the response to both requests.
>>>
>>> Option 3 -- Respond to A, then trigger another replication for B
>>> In this case we wait till the replication started by A has  
>>> completed, then
>>> do an incremental one and respond to B with the results of that  
>>> incremental.
>>>
>>> I think I'd vote for 3.  Cheers, Adam
>>>
>>>
>>
>> If I follow this correctly, the issue is, "POST to _replicate, a
>> second POST to _replicate occurs before the first request finishes"
>> (with the same source/target info).
>>
>> My knowledge of replication is only cursory, but I could also see:
>>
>> Option 4:
>>
>> Same as views, we wait for replication to finish and return the same
>> result to all clients that made a request.
>
> I understand this and Adam's option 3 to be the same. What am I  
> missing? :)

No, not quite.  In Option 3 the two requesters get different  
responses.  A gets the result of the original request, B gets the  
result of the replication triggered automatically after the first one  
that replicates any updates to the DB which happened during the first  
pass.  If no updates occurred, B will receive the result of the first  
replication.

Paul's Option 4 is more like Options 1 and 2, where A and B get  
identical responses.  The difference between 1 and 2 is just whether  
new updates get included in that response.

Whew.

>> Option 5:
>>
>> Return an error on B that says, "Yeah, yeah. Already on it."
>
> This would make replication behave a bit like compaction.

Sort of, in that additional triggers are no-ops.  Option 1 also has  
that behavior.

> I think I like 3/4 best.
>
> Cheers
> Jan
> --


Best, Adam