Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of jchris@gmail.com designates
 74.125.83.180 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=dnNVfSj3uAy9DIl5IU2Z13sMPtVkKmTyQPIylQUsqm9vfJxml9rxRTQFYf5GWl7nNZ
         6+YCaO1IFKg//WDvrmSKZklxoh0VkCTqeOKxRTSEAwQWyShpzH2K9faDKTJreYQvdcfZ
         mfkCQxdk8dia+5ELWFnFQocxyxlV1C1CgQkbk=
MIME-Version: 1.0
Sender: jchris@gmail.com
In-Reply-To: <EFE733F8-7CAF-4317-A138-968E2142D303@apache.org>
References: <ea2499da1003050808s2cd953cfn3b226ef1ee501330@mail.gmail.com>
	 <E46BEF20-5D8A-485B-AA56-40D4C96F4F35@gmail.com>
	 <AB172C32-EA65-44A7-83D3-0EF51D816192@apache.org>
	 <ec7a93a1003051113q62e2128mfcd916d6615a5490@mail.gmail.com>
	 <EFE733F8-7CAF-4317-A138-968E2142D303@apache.org>
Date: Fri, 5 Mar 2010 14:01:10 -0800
Message-ID: <e282921e1003051401h24344ebdud8dbfb4e7203541@mail.gmail.com>
Subject: Re: Preserving seq order through replication
From: Chris Anderson <jchris@apache.org>
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Mar 5, 2010 at 11:27 AM, Adam Kocoloski <kocolosk@apache.org> wrote=
:
> On Mar 5, 2010, at 2:13 PM, Randall Leeds wrote:
>
>> I believe replication right now sorts the incoming documents by seq
>> (comment says smth like 'just in case'), but then they are fetched
>> with some amount of concurrency and inserted as they arrive. Adam,
>> please chime in if I'm reading it wrong, as I think some of those
>> comments are yours.
>
> Yep, that's basically it.
>
>> On Fri, Mar 5, 2010 at 09:10, Adam Kocoloski <kocolosk@apache.org> wrote=
:
>>> With that said, making replication preserve the update order is probabl=
y not very difficult or expensive to do. =A0Best,
>>>
>>> Adam
>>
>> It could be a performance loss if couch_rep_writer had to buffer
>> writes to preserve insertion order. Alternatively, to prevent the
>> write queue from growing unboundedly if one document repeatedly fails,
>> couch_rep_reader could wait for a chunk of contiguous documents before
>> handing them over to the writer.
>
> If we were to do this, I'd implement it that 2nd way, where the reader on=
ly hands over contiguous blocks to the writer, and doesn't "get too far ahe=
ad of itself"

I still think the whole notion is unnecessary complexity that creates
guarantees we'd rather not have. But I'm not gonna say don't write it.
It's just that if someone relies on this, we'll have to do extra work
to explain to them why their code broke when they scaled up.

_local_seq should be considered a smell (but sometimes a necessary one
for realtime apps...)

>
>> Concurrently fetching documents rather than just pipelining them on
>> one http connection might not seem beneficial at first glance, but a
>> source which has disks that can service concurrent reads stands to
>> benefit. When the source can't actually do this it's up to the
>> OS/FS/Disk to order reads in a way that is as optimal as possible.
>
> +1
>
> Adam


--=20
Chris Anderson
http://jchrisa.net
http://couch.io