couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antony Blakey <antony.bla...@gmail.com>
Subject Re: Architectural advice requested, and some 1.0 suggestions.
Date Thu, 06 Nov 2008 04:12:27 GMT

On Tue, Nov 4, 2008 at 6:47 PM, Antony Blakey  
<antony.blakey@gmail.com> wrote:

> My solution is to use the notification mechanism to maintain  
> (multiple)
> SQLite databases containing the document keys I need to search over.  
> In the
> URP example, I store (db, src, dest) records. I also store the last  
> seqno,
> so that I can do incremental updates to SQLite.
>
> Then, using _external, I can make queries of SQLite to get me either  
> (user,
> permission) pairs using a self-join, or in the case of the arbitrary
> metadata queries, a list of document ids.
>
> The primary difficulties I have with this simple model are:
>
> 1) The notification mechanism doesn't give me enough information.  
> Currently
> I have to do a _all_docs_by_seq and check for deletions by  
> attempting to get
> each document, which I have to do for every document in any case  
> (unless I
> use transparent id's) to determine if I'm interested in it, and then  
> get the
> data. I presume this technique works because deletion is actually a  
> form of
> update until compaction (I copied it from GeoCouch).
>
> ** SUGGESTION ** I would prefer an update interface that gave me a  
> stream of
> (old, new) document pairs, which covers add/update/delete, plus a  
> (from, to)
> seqno pair. Have a 'from' seqno lets me know when I have to trigger  
> a full
> rescan, which I need to do in a variety of circumstances such as
> configuration change.

In retrospect this is not a good idea. I think notification handlers  
should do nothing more than mark a view or query source as dirty,  
invalidate a cache such as memcached, and possibly check for mods to a  
config document to enable/disable the query service. The _external/ 
plugin query handler should do the subsequence processing and update  
any private data structures or on-disk indexes, just as the map/reduce  
views do, and for the same reason. So I don't think the notification  
mechanism should be changed.

However, that raises a question about external view/query server  
updating: should a view update to the seqnum that is current when the  
request is received, or should it keep looping in an update cycle  
until the seqnum has reached a steady state?

The former would make sense if you just wanted the ensure that the  
view was up-to-date with records a client might have just written in  
the requesting thread, whilst the later would seem to potentially  
block forever depending on the amount of processing required to update  
the external view and the update rate.

Finally, does anyone have advice about the merits of mnesia vs. sqlite  
(or another sql db) for this kind of auxiliary indexing?

Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

Lack of will power has caused more failure than lack of intelligence  
or ability.
  -- Flower A. Newhouse


Mime
View raw message