incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Jacob" <a...@hjksolutions.com>
Subject Re: Is it possible to evaluate a view on a 20.000 documents database?
Date Fri, 01 Aug 2008 16:50:54 GMT
On Fri, Aug 1, 2008 at 12:37 AM, Johan Liseborn
<johan.liseborn@gmail.com> wrote:
> AFAIU, when you add new documents and then evaluate a view including
> those documents, indexing will happen, but only for the newly added
> documents (i.e. already indexed documents will not be re-indexed). I
> believe this means that the time to index will be, in some way,
> proportional to the number of *new* documents. I believe I have seen a
> big-O "number" for this somewhere, but I don't remember right now if
> it is O(n), O(log n), or something else (I am sure someone else on the
> list can answer that :-).
>
> As can be seen from the results, when CouchDB had to index the 10.000
> new documents, it took about 13 minutes to get the result, but when
> all the documents had been indexed, the answer came back in 0.7
> seconds. Having to index 10 documents did not take that long, giving
> an answer in 1.2 seconds.

I did some totally off-the-cuff benchmarking with Varnish (a caching
HTTP reverse proxy) in front of CouchDB.  It works entirely as
expected - Varnish returns lightning fast (<30ms) very consistently
(20ms in the 50%, 35ms in the longest.)

With Varnish, you can also add PURGE support for a given URL.

This means that, if those compute times are egregious, and you can
suffer your data being a bit out of date, it would be trivial to
create a system that does:

Reads...
User -> Varnish -> CouchDB

Writes...
User->Varnish->CouchDB->PURGE for entry->Queue for view
re-indexing->PURGE for views

Essentially ensuring that all your reads always return as quickly as
possible, masking any possible delay in index queries on bulk updates
(assuming stale data for 10 minutes is ok.)

Regards,
Adam

-- 
HJK Solutions - We Launch Startups - http://www.hjksolutions.com
Adam Jacob, Senior Partner
T: (206) 508-4759 E: adam@hjksolutions.com

Mime
View raw message