Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAAL6JQiUVVdpRuDrsUHQOBThO=rXQ6qiEDr_2hpdg-7RvSODnA@mail.gmail.com>
References: 
 <2073127356.24602.1324243590757.JavaMail.tomcat@hel.zones.apache.org>
 <1765975633.44958.1324877790680.JavaMail.tomcat@hel.zones.apache.org>
 <CAN-3CBJpRQ3ya61YOuObunwa2uxiVAvmQCadMtU5me0KAqUpdQ@mail.gmail.com>
 <F740EBE2-020D-4B18-A126-D34C8AEDECCD@dionne-associates.com>
 <CAN-3CB+U3a8=sfE-PZo6pCpO+por7Dt_jZQasNbpCY+xU1-cqg@mail.gmail.com>
 <CAAL6JQj4g761nJ829Xj_pz_pUpM5r0KwQWBPhJMKywPvoHBO7g@mail.gmail.com>
 <CAJNb-9ooAjh1aC=tWgfcXYfdgZd1haxceeKTOtnDBXiZtGUX7A@mail.gmail.com>
 <CAAL6JQi+6oau5ozHKYYnYt5ATVfeivF4SG3W4htqGUgVF+Yt0g@mail.gmail.com>
 <CAN-3CBLYqh6CTF2x=Z66CshKm2FSofVHpAhz0akoW4SVz4kHyQ@mail.gmail.com>
 <CAAL6JQiUVVdpRuDrsUHQOBThO=rXQ6qiEDr_2hpdg-7RvSODnA@mail.gmail.com>
From: Jason Smith <jhs@iriscouch.com>
Date: Wed, 28 Dec 2011 10:37:02 +0700
Message-ID: 
 <CAN-3CBKRBoHGi11VqY+9HyXJ8O=ANEeqo8hm4wJVwSn3b4DXcw@mail.gmail.com>
Subject: Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db
 - the db increases its update_seq counter when viewing stats - but not when
 getting changes
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=UTF-8

On Wed, Dec 28, 2011 at 9:38 AM, Randall Leeds <randall.leeds@gmail.com> wrote:
> On Tue, Dec 27, 2011 at 05:22, Jason Smith <jhs@iriscouch.com> wrote:
>> On Tue, Dec 27, 2011 at 5:04 AM, Randall Leeds <randall.leeds@gmail.com> wrote:
>>> Either
>>> these things have a proper place in the sequential history of a
>>> database or they do not. That there are things which affect update_seq
>>> but do not appear in the by_seq index and _changes feed feels like a
>>> mistake.
>>
>> The first sentence is, well, a tautology actually, but it asks the
>> right question and the answer is they DO NOT belong. _changes shows
>> data, not metadata. By definition, _changes is anything worth
>> replicating.
>
> That strikes me as incorrect. The _changes feed is purely metadata
> unless ?include_docs=true is specified.

Yes, "data" and "metadata" are problematic words. I'll stop using them.

Do you agree that _changes is, by definition, anything worth replicating?

>> But I hope my filesystem example above shows why it is okay to
>> increment update_seq but not change by_seq.
>
> You show a nice precedent for separating metadata and data, but
> CouchDB has a decent precedent of avoiding this same thing. For
> example, _id and _rev are in the returned document body rather than
> part of the HTTP request (it could have been just URL and entity tag
> headers only for this).

Yeah that's a good point.

>> The bug with update_seq is not that it it is too eager (increments for
>> _security, _revs_limit), but it is not eager enough (it should bump
>> for _local too).
>>
>
> I agree, but for different reasons. I think _local docs may have a
> place in by_seq even if the default _changes request still only shows
> the default, replicable documents.

That's an interesting idea.

IMO, _security, _revs_limit, apply to a specific database and URL, and
consequently must never replicate. _local docs are those which don't
replicate. If _local would replicate, I'd worry about spurious
checkpoints spreading to where they don't belong; and unchecked
_security replication is even worse.

Your idea improves consistency and orthogonality. It also solves the
problem of how to enumerate _local docs. (AFAIK there is no way to
list them all, not via _all_docs, or _changes, or a view).

But it doesn't solve the larger problem: How to follow a _changes feed
and know when you have caught up. Both Bob N. and I independently did
the following for our projects:

1. GET /db and wrongly assume update_seq will appear in the changes feed
2. GET /db/_changes?feed=continuous
3. Break when a change has .seq >= update_seq

Suppose you have step 0: Update _security or _revs_limit. The loop
will never break.

You propose (WLOG) _changes?comprehensive=true which guarantees a
change equal or greater than update_seq. That's cool, but IMO app
developers now have to add code to ignore irrelevant changes like
those containing replication checkpoints.

I propose (WLOG) update_sikh in the db header which is the seq id of
the latest *document* update. App developers modify their step 1 to
use update_sikh instead of update_seq.

Is that an accurate synopsis?

>> 2. As a frequent consumer of _changes, I would prefer *not* to see
>> _local documents, nor _security or other updates in there. They are
>> metadata, not data. Maybe I misunderstood, but nobody wants to
>> *replicate* _security objects or _local docs; they just want MVCC
>> semantics (Adam on _security, IIRC) and a simplified API (me, on
>> making all metadata a _local doc, and making _local docs full MVCC).
>
> I think you misunderstand, maybe. In the case of BigCouch, MVCC is all
> that's needed because the replication does not go over HTTP. I see no
> reason to require that special care be taken to copy these objects
> when a flag on the _changes feed might cause them to be transferred
> very naturally. In particular, I would use this feature in a
> hypothetical Lounge 3.0. It also means that with admin privileges we
> could do full backup replications.

If couch could do this, then cool. But consider that both examples are
the same special-case: sharding and simulating a normal database API
when there are actually multiple parts. That sounds like an
application concern.

Is it really true that shards need the same _revs_limit as the
simulated whole? Maybe they really want _revs_limit /
number_of_shards?

Is it really necessary that _security be identical in each shard?
Actually, yes it is, because validate_doc_update uses it. But still...

How are you computing doc_count in the /db response? You have to sum
doc_count from each shard. But every shard needs a copy of every
design doc for validation. So you have to subtract those back out? My
broader point is, sharding applications already do lots of magic. I'm
not sure if replicating _security and _local docs buys you much.

But you've definitely persuaded me that your idea works. It is the
second-best proposal I've seen in this thread :)

-- 
Iris Couch