couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Shumaker <sshuma...@gmail.com>
Subject Re: View Performance (was Re: The 1.0 Thread)
Date Thu, 02 Jul 2009 21:15:47 GMT
We see times that are considerably worse.  We mostly have maps - very
few reduces.  We have 40k objects, about 25 design docs, and 90 views.
 Although we're about to change the code to auto-generate the design
docs based on the view filters used (re: view filter patch) - see if
that helps.

Maybe it's because we have larger objects - but re-indexing a typical
new view takes > 5 minutes (with view filtering off).  Some are worse.
 With view filtering on some can be quite fast - some views finish in
like 10 seconds.  Interestingly, reindexing all views takes about an
hour - with or without view filtering.  I'm guessing that a
substantial part of the bottleneck is erlang -> json serialization.
Many of our objects are heavily nested structures and exceed 10k in
size.  One other note - when we tried dropping in the optimized
'main.js' posted on the mailing list, we saw an overall 20% speedup.
Unfortunately, it wasn't compatible with the authentication stuff, and
the deployment was a bit wacky, so we're holding off on that right
now.


On Thu, Jul 2, 2009 at 11:30 AM, Damien Katz<damien@apache.org> wrote:
>
> On Jul 2, 2009, at 1:55 PM, Paul Davis wrote:
>
>> On Thu, Jul 2, 2009 at 1:29 PM, Damien Katz<damien@apache.org> wrote:
>>>
>>> On Jul 2, 2009, at 1:16 PM, Jason Davies wrote:
>>>
>>>> On 2 Jul 2009, at 15:38, Brian Candler wrote:
>>>>
>>>>> For some fruit that was so low-hanging that I nearly stubbed my toe on
>>>>> it,
>>>>> see https://issues.apache.org/jira/browse/COUCHDB-399
>>>>
>>>>
>>>> Nice work!  I'd be interested to see what kind of performance increase
>>>> we
>>>> get from Spidermonkey 1.8.1, which comes with native JSON
>>>> parsing/encoding.
>>>>  See here for details:
>>>> https://developer.mozilla.org/En/Using_native_JSON .
>>>>
>>>> Rumour has it 1.8.1 will be released any time soon (TM)
>>>
>>> I'm not sure the new engine is such a no-brainer. One thing about the new
>>> generation of JS VMs is we've seen greatly increased memory usage with
>>> earlier versions. Also the startup times might be longer, or shorter.
>>>
>>> Though I wonder if this can be improved by forking a JS process rather
>>> than
>>> spawning a new process.
>>>
>>
>> Memory usage is a definite concern. I'm not sure I follow why startup
>> times would be important though. Am I missing something?
>
> Start up time isn't a huge concern, but it's is a something to consider. On
> a heavily loaded system, scripts that normally work might start to time out,
> requiring restarting the process. Lots of restarts may start to eat lots cpu
> and memory IO.
>
> -Damien
>
>
>>
>>> -Damien
>>>
>>>> --
>>>> Jason Davies
>>>>
>>>> www.jasondavies.com
>>>>
>>>
>>>
>
>

Mime
View raw message