couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Davis" <paul.joseph.da...@gmail.com>
Subject Re: view index build time
Date Wed, 02 Jul 2008 23:19:05 GMT
Are there any plans on making this parallel in the future? Splitting
up the docs amongst a set of processes and having them sort local
copies before doing a merge sort back into the main index file doesn't
seem conceptually hard.

Also, where do the javascript conversions happen? Wouldn't that be in
the beam process with mochiweb?

On Wed, Jul 2, 2008 at 6:28 PM, Chris Anderson <jchris@grabb.it> wrote:
> On Wed, Jul 2, 2008 at 3:08 PM, Paul Davis <paul.joseph.davis@gmail.com> wrote:
>> I'd have to go back and double check, but off the top of my head 25
>> min for 300K docs seems about like what I was getting. Ie, not orders
>> of magnitude slower or anything.
>
> In my experience, views generate about 1/2 as fast as that, if not
> more slowly. My views are often quite complex with a lot of internal
> looping and multiple emits, so that probably explains it. In short,
> the times you're reporting seem reasonable.
>
> The bottleneck (based on my extremely unscientific use of top) doesn't
> seem to be the view server, but rather CouchDB's beam process, which
> as I understand it, is busy sorting the results as they come back from
> the view server. So the quickest route to parallelizing this may be to
> manually partition your data across CouchDB instances, generate the
> views, and query them in parallel, merging the results in your
> application.
>
> I don't actually plan to do all that work until my insert rate
> eclipses CouchDB's view generation speed. :)
>
> Once upon a time there was a feature to return the available results
> of a view, even while generation is still occurring. The feature has
> fallen by the wayside, and it would be non-trivial to turn it back on,
> according to Damien on IRC. Maybe if it would be useful to enough
> people, we'll see it again.
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>

Mime
View raw message