couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kowsik <kow...@gmail.com>
Subject Re: Question on view server design...
Date Fri, 24 Oct 2008 16:32:56 GMT
Agreed on the armchair philosophy. Given the huge potential for couch,
just trying to see how far we can push it from a scalability
perspective.

On a related note, are there thoughts on how best to organize the
design views? Given how the couchjs <=> erlang interaction works, what
I'm finding is that if I have a number of map/reduce tuples in a given
design view, the "map_doc" command returns keys generated by all of
the map functions and sometimes there's 100's of KBytes (sometimes
MBytes) of JSON data exchanged between the two processes. If I split
them up, then each design view (with now just one map/reduce tuple)
runs pretty fast and there's no thrashing. I know it's too early to
talk about "best practices", but what's the "recommended" way of
organizing design views?

Thanks,

K.

On Thu, Oct 23, 2008 at 10:15 AM, Jan Lehnardt <jan@apache.org> wrote:
>
> On Oct 23, 2008, at 19:08, kowsik wrote:
>
>> No, not explicitly, but I was looking at evalcx which sets up and
>> tears down a new JSContext each time a function is compiled. Just
>> wondering if there's an incremental way of doing this so we don't
>> incur the overhead each time reduce is invoked.
>
> Armchair optimization is futile, sorry. Without the number to back it up,
> it doesn't make any sense wo change this.
>
>> Maybe with TraceMonkey, this is a moot point? ;-)
>
>
> Tracemonkey is not there yet, initial tests showed promising execution
> speeds, but less than stellar memory usage. I expect to see improvements
> in this area, though. It'll be good! :)
>
> Cheers
> Jan
> --
>
>>
>>
>> K.
>>
>> On Thu, Oct 23, 2008 at 9:42 AM, Jan Lehnardt <jan@apache.org> wrote:
>>>
>>> On Oct 23, 2008, at 18:35, kowsik wrote:
>>>
>>>> I was seeing the couchjs process getting pegged and I was looking into
>>>> further.
>>>>
>>>> 1. The protocol uses "add_fun" command to add all the map functions to
>>>> the view server so that they are compiled and ready to go.
>>>> 2. Then "map_doc" is invoked with each document to return the results
>>>> of the various emit's
>>>> 3. They "reduce" and "rereduce" are invoked depending on the query.
>>>>
>>>> It's #3 that I'm interested in. The "reduce" command currently passes
>>>> in the reduce function string from the design view each time it's
>>>> invoked. This means the reduce function is first eval'd and then
>>>> executed. I'm wondering if it's possible to rename "add_fun" to
>>>> "add_map_fun" and introduce a new "add_reduce_fun" and then reference
>>>> functions by index? Seems like it might speed things up quite a bit.
>>>
>>> Have you measured the reduce function compilation to be a bottleneck?
>>> I'd guess (see, haven't measured either:-) that erlang-term to JSON &
>>> JSON
>>> parsing take up significantly more time.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>
>
>

Mime
View raw message