incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Updating views on save
Date Mon, 28 Apr 2008 08:22:03 GMT
Heya Sebastian,
it seems you feel rather strongly about this issue. But that's
nothing a little engineering can solve for you, read on :)

On Apr 28, 2008, at 01:04, Guby wrote:
> Hello dear Couchers
>
> I understand that the views are indexed the first time they are  
> accessed and as far as I know there is no way to turn on view  
> updating on document save. I really don't understand the reasoning  
> behind this behavior. The advantage of the pre-populated/indexed  
> views are that they are blazingly fast to query and access, but that  
> advantage disappears when the first request after a document update  
> has to regenerate the view first!
> I am currently building a web app where the background processes  
> perform a lot of writes to the database. The time it takes to write  
> a document is not critical for me. What is critical though is the  
> time it takes to load web pages for the end user that require  
> content from the database.
> In some situations the background processes add thousands of  
> documents to the database within a short period of time, and when  
> the user tries to access a page after such an update the view  
> querying sometimes takes minutes and as a consequence of that the  
> browser times out... Not a recipe for happy customers...
>
> The only solution I can see at the moment is to create a worker that  
> queries the database whenever it is told that there has been a  
> document update, but that seems really stupid and unnecessary. And  
> in my case, running on a smallish VPS, a big waste of resources  
> having an extra working doing something the database itself could  
> just as well have done. It also requires a lot of extra coding  
> notifying the worker whenever I update or create a document  
> throughout my app.

That would be a rather extreme solution. Why not, for
example, trigger a view update from your document-
insertion code, every N (N = 10, 30, 60?) seconds?


> I am sure you have reasons for having implemented the views the way  
> you have, but I would be really interested to hear why it has been  
> done this way!

1) To not have a 'write penalty' for all views when
documents are added. We expect you to have
quite a few of views and updating all of them on-write
seems silly. The data is generated when needed,
saving resources by 2) not clogging them up when
needed elsewhere and 3) processing large quantities
of data in batches. and finally 4) The very layout of the
bytes that make up documents on disk and the way they
are read are optimised for super-fast index creation. This
is expected to be a common operation. I still understand
that this leaves things to be desired for you.


> My wishes are for an optional updating of views on save feature! In  
> some cases that might regenerate a view several times without it  
> actually being accessed in between, but that is a tradeoff I can  
> live with, slow views on the other hand is something I can not!

Put this in a shell script called view_trigger.sh

#!/bin/sh

counter=0
max_docs=100

while true
	do
	read database

	counter=`expr $counter + 1`

	if [ $counter -ge $max_docs ]; then
		`curl http://server:5984/$database/_view/name?count=0`
		counter=0
	fi
done

and add view_trigger.sh to our couch.ini as a  
DbUpdateNotificationProcess

voilá :)

Yes, this is extra work externally, but this is still a sensible
solution. From our perspective, we do not need to change
the core server behaviour to get you what you need and
you still benefit from the batching of index creation.

Also, I'd like to second what Cortland said: All views in a
design document get updated if you query one of them.
Be aware of that :)

And on a final note: Thanks for writing in. Don't be
discouraged by the replies. If there are other things that
you would love to see in CouchDB, please let us know.

Also, if enough users request a feature, we will consider
putting it in, even on-write view updates, but do not expect
that to happen anytime soon or at all. There are a lot
of things that need our attention first.

Cheers
Jan
--
Mime
View raw message