couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien Katz <dam...@apache.org>
Subject Re: Map/Reduce takes lots of time every request
Date Sun, 23 Nov 2008 12:58:06 GMT
It sounds like ever you are using a temp view, or an older version of  
Erlang (R11 I think) that has performance problems.

-Damien

On Nov 22, 2008, at 4:20 PM, maddiin wrote:

> > It sounds like you are building a tag-cloud.
>
> That is right.
>
>
> > I'm curious how long it takes with reduce=false
>
> With reduce=False it takes around 50 seconds to load all tags.
> 1000 tags are loading in 2-3 seconds with reduce=False.
>
> With reduce=True it takes around 80 seconds to load all tags.
> 1000 tags are loading in 8-9 seconds with reduce=True.
>
>
> > The smart money would be on caching the results of that operation,  
> which is standard practice with SQL based tag clouds as well.
>
> I guess I will have to unless..
>
> > Also, I'm not sure, but perhaps it would be possible for CouchDB  
> to cache final reduce values in the btree as well, so that  
> group=true queries can save the cost of the final rereduce (and make  
> subsequent queries fast...).
>
> That'd be nice.
>
>
> Thanks for the response,
> maddiin
>
>
> Chris Anderson schrieb:
>> On Sat, Nov 22, 2008 at 12:25 PM, maddiin <maddiin@googlemail.com>  
>> wrote:
>>
>>> Do you have any advice what I am doing wrong and how I could speed  
>>> this up?
>>>
>>
>>
>> I'm curious how long it takes with reduce=false (should be limited
>> basically by IO).
>>
>> I'm almost certain (please correct me if I'm wrong) that reduce
>> requests must call the JavaScript interpreter at least once per
>> request, to rereduce the btree inner-nodes that fit in that request
>> range. This means for group=true requests, the rereduce function must
>> run once per unique key (at minimum). That would be the source of  
>> your
>> slowness. It sounds like you are building a tag-cloud. The smart  
>> money
>> would be on caching the results of that operation, which is standard
>> practice with SQL based tag clouds as well.
>>
>> If you're not doing a tag cloud, maybe there's a way you can get the
>> needed results using map only?
>>
>> Also, I'm not sure, but perhaps it would be possible for CouchDB to
>> cache final reduce values in the btree as well, so that group=true
>> queries can save the cost of the final rereduce (and make subsequent
>> queries fast...)
>>
>> Chris
>>
>>
>>
>



Mime
View raw message