couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Practical storage limit
Date Sun, 13 Jul 2008 16:46:46 GMT

On Jul 13, 2008, at 18:05, Peter Eddy wrote:

>> What I'd do is query the view for updating after each bulk insert.
>
> This is what I've been doing, however it seems less than ideal. It
> means that the bulk insert code needs to know about all views that
> have been defined.
>
> What I want to avoid, of course, is a user invoking a view and having
> to wait a long time (minutes in some cases) for that view to be
> updated in order to get their results. I would gladly trade document
> insertion time for automatic view updates.
>
> It could be argued that bulk updates are a special case that happen
> only in exceptional situations. But I can easily imagine normal
> application functionality that might add many documents, and this code
> would then need to think about how many documents were added and if
> the views should be updated in the process in order to avoid long
> user-time delays. The larger the application, the more messy and error
> prone this becomes.
>
> Anyway, I've been meaning to mention this for a long time, so since it
> came up...

I'll make a FaQ out of this :-)

To get on write view update semantics, you can write a little daemon
script that runs alongside CouchDB and is specified in couch.ini
as DbUpdateNotificationProcesses. This deamon gets sent a
notification each time the database is changed and could in turn
trigger a view update every N document inserts or every Y seconds,
whichever occurs first. The reason not to integrate each doc as
it comes in is that it is horribly inefficient and CouchDB is designed
to do view index updates very fast, so batching is a good idea.

To get a list of all views in a database, you can do a
GET /db/_all_docs?startkey=_design/&endkey=_design/ZZZZ
(we will have a /db/_all_design_docs view to make the ZZZZ-hack
go away).

That should solve your problem.

Yes, such a daemon should be shipped with CouchDB, but we
haven't got around to work on the deployment infrastructure yet.
Any contributions to this are very welcome. I think the developer's
choice of language for helper scripts is Python, but any will do,
whatever suits you best.

Cheers
Jan
--
The FaQ entry is at the bottom of this page.
http://wiki.apache.org/couchdb/FrequentlyAskedQuestions
BTW: Is there a FaQ module or whatever for MoinMoin? it
would be nice to get a MediaWiki-like table of contents at
the top. Noah?

Mime
View raw message