Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <54390D26-2473-4B42-A2F2-EBBC7ED91D5A@apache.org>
References: <EBDEACDE-17FD-4DD5-8E67-D5A13D69F88E@apache.org>
 <CAN-3CBLK=GywcRePcaOt_c4b1MhNZ+7aoNLLJ6XqFQKeth17wg@mail.gmail.com>
 <54390D26-2473-4B42-A2F2-EBBC7ED91D5A@apache.org>
From: Jason Smith <jhs@iriscouch.com>
Date: Wed, 17 Aug 2011 10:15:32 +0700
Message-ID: 
 <CAN-3CBLGcnEiFvPPoPigEEf-Kgx-eWvo3AC1qA2KU-CAgdvmWw@mail.gmail.com>
Subject: Re: The replicator needs a superuser mode
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Wed, Aug 17, 2011 at 9:49 AM, Adam Kocoloski <kocolosk@apache.org> wrote=
:
> On Aug 16, 2011, at 10:31 PM, Jason Smith wrote:
>
>> On Tue, Aug 16, 2011 at 9:26 PM, Adam Kocoloski <kocolosk@apache.org> wr=
ote:
>>> One of the principal uses of the replicator is to "make this database l=
ook like that one". =C2=A0We're unable to do that in the general case today=
 because of the combination of validation functions and out-of-order docume=
nt transfers. =C2=A0It's entirely possible for a document to be saved in th=
e source DB prior to the installation of a ddoc containing a validation fun=
ction that would have rejected the document, for the replicator to install =
the ddoc in the target DB before replicating the other document, and for th=
e other document to then be rejected by the target DB.
>>
>> Somebody asked about this on Stack Overflow. It was a very simple but
>> challenging question, but now I can't find it. Basically, he made your
>> point above.
>>
>> Aren't you identifying two problems, though?
>>
>> 1. Sometimes you need to ignore validation to just make a nice, clean co=
py.
>> 2. Replication batches (an optimization) are disobeying the change
>> sequence, which can screw up the replica.
>
> As far as I know the only reason one needs to ignore validation to make a=
 nice clean copy is because the replicator does not guarantee the updates a=
re applied on the target in the order they were received on the source. =C2=
=A0It's all one issue to me.
>
>> I responded to #1 already.
>>
>> But my feeling about #2 is that the optimization goes too far.
>> replication batches should always have boundaries immediately before
>> and after design documents. In other words, batch all you want, but
>> design documents [1] must always be in a batch size of 1. That will
>> retain the semantics.
>>
>> [1] Actually, the only ddocs needing their own private batches are
>> those with a validate_doc_update field.
>
> My standard retort to transaction boundaries is that there is no global o=
rdering of events in a distributed system. =C2=A0A clustered CouchDB can tr=
y to build a vector clock out of the change sequences of the individual ser=
vers and stick to that merged sequence during replication, but even then th=
e ddoc entry in the feed could be "concurrent" with several other updates. =
=C2=A0I rather like that the replicator aggressively mixes up the ordering =
of updates because it prevents us from making choices in the single-server =
case that aren't sensible in a cluster.

That is interesting. So if it is crucial that an application enforce
transaction semantics, then that application can go ahead and
understand the distribution architecture, and it can confirm that a
ddoc is committed and distributed among all nodes, and then it can
make subsequent changes or replications.

Or, written as a dialogue:

Developer: My application knows or cares that Couch is distributed.
Developer: My application depends on a validation function applying univers=
ally.
Developer. But my application won't bother to confirm that it's been
fully pushed before I make changes or replications.
Adam: WTF?

Snark aside, it's an excellent point. Thanks.

--=20
Iris Couch