couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Question on view server design...
Date Fri, 24 Oct 2008 16:47:58 GMT
Hi,

On Oct 24, 2008, at 18:32 , kowsik wrote:

> Agreed on the armchair philosophy.

Cool! :) One thing though…

> Given the huge potential for couch,
> just trying to see how far we can push it from a scalability
> perspective.

... nit-picking. Single code path execution speed has little to do
with scaling :)


> On a related note, are there thoughts on how best to organize the
> design views? Given how the couchjs <=> erlang interaction works, what
> I'm finding is that if I have a number of map/reduce tuples in a given
> design view, the "map_doc" command returns keys generated by all of
> the map functions and sometimes there's 100's of KBytes (sometimes
> MBytes) of JSON data exchanged between the two processes. If I split
> them up, then each design view (with now just one map/reduce tuple)
> runs pretty fast and there's no thrashing. I know it's too early to
> talk about "best practices", but what's the "recommended" way of
> organizing design views?

Trade-offs.

Consider two views A and B. If they reside in the same design document,
both are updated if you query either. Docs have to be converted to JSON
and back to JS only once for each view. If they live in separate design
documents, each doc has to be converted twice; once for each view.

Grouping views in design documents makes better use of your CPU cycles.
If one view in a group takes particularly long to update, you might  
want to
put that into a separate design document to speed up requests to your  
other
views.

Design documents are designed (heh) to hold all information (even code)
for a single application. Having all views for an application in a  
single design
document follows this philosophy. There are valid cases to break this  
rule
though. Note that your application will be not as simple to handle.


Cheers
Jan
--


> On Thu, Oct 23, 2008 at 10:15 AM, Jan Lehnardt <jan@apache.org> wrote:
>>
>> On Oct 23, 2008, at 19:08, kowsik wrote:
>>
>>> No, not explicitly, but I was looking at evalcx which sets up and
>>> tears down a new JSContext each time a function is compiled. Just
>>> wondering if there's an incremental way of doing this so we don't
>>> incur the overhead each time reduce is invoked.
>>
>> Armchair optimization is futile, sorry. Without the number to back  
>> it up,
>> it doesn't make any sense wo change this.
>>
>>> Maybe with TraceMonkey, this is a moot point? ;-)
>>
>>
>> Tracemonkey is not there yet, initial tests showed promising  
>> execution
>> speeds, but less than stellar memory usage. I expect to see  
>> improvements
>> in this area, though. It'll be good! :)
>>
>> Cheers
>> Jan
>> --
>>
>>>
>>>
>>> K.
>>>
>>> On Thu, Oct 23, 2008 at 9:42 AM, Jan Lehnardt <jan@apache.org>  
>>> wrote:
>>>>
>>>> On Oct 23, 2008, at 18:35, kowsik wrote:
>>>>
>>>>> I was seeing the couchjs process getting pegged and I was  
>>>>> looking into
>>>>> further.
>>>>>
>>>>> 1. The protocol uses "add_fun" command to add all the map  
>>>>> functions to
>>>>> the view server so that they are compiled and ready to go.
>>>>> 2. Then "map_doc" is invoked with each document to return the  
>>>>> results
>>>>> of the various emit's
>>>>> 3. They "reduce" and "rereduce" are invoked depending on the  
>>>>> query.
>>>>>
>>>>> It's #3 that I'm interested in. The "reduce" command currently  
>>>>> passes
>>>>> in the reduce function string from the design view each time it's
>>>>> invoked. This means the reduce function is first eval'd and then
>>>>> executed. I'm wondering if it's possible to rename "add_fun" to
>>>>> "add_map_fun" and introduce a new "add_reduce_fun" and then  
>>>>> reference
>>>>> functions by index? Seems like it might speed things up quite a  
>>>>> bit.
>>>>
>>>> Have you measured the reduce function compilation to be a  
>>>> bottleneck?
>>>> I'd guess (see, haven't measured either:-) that erlang-term to  
>>>> JSON &
>>>> JSON
>>>> parsing take up significantly more time.
>>>>
>>>> Cheers
>>>> Jan
>>>> --
>>>>
>>>
>>
>>
>


Mime
View raw message