couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Shumaker <sshuma...@gmail.com>
Subject Re: View Performance (was Re: The 1.0 Thread)
Date Thu, 02 Jul 2009 21:24:37 GMT
CouchDB: 0.9.0
Erlang - r12b.

On Thu, Jul 2, 2009 at 2:17 PM, Paul Davis<paul.joseph.davis@gmail.com> wrote:
> On Thu, Jul 2, 2009 at 5:15 PM, Scott Shumaker<sshumaker@gmail.com> wrote:
>> We see times that are considerably worse.  We mostly have maps - very
>> few reduces.  We have 40k objects, about 25 design docs, and 90 views.
>>  Although we're about to change the code to auto-generate the design
>> docs based on the view filters used (re: view filter patch) - see if
>> that helps.
>>
>> Maybe it's because we have larger objects - but re-indexing a typical
>> new view takes > 5 minutes (with view filtering off).  Some are worse.
>>  With view filtering on some can be quite fast - some views finish in
>> like 10 seconds.  Interestingly, reindexing all views takes about an
>> hour - with or without view filtering.  I'm guessing that a
>> substantial part of the bottleneck is erlang -> json serialization.
>> Many of our objects are heavily nested structures and exceed 10k in
>> size.  One other note - when we tried dropping in the optimized
>> 'main.js' posted on the mailing list, we saw an overall 20% speedup.
>> Unfortunately, it wasn't compatible with the authentication stuff, and
>> the deployment was a bit wacky, so we're holding off on that right
>> now.
>>
>
> What version of Erlang and CouchDB are you using?
>
>>
>> On Thu, Jul 2, 2009 at 11:30 AM, Damien Katz<damien@apache.org> wrote:
>>>
>>> On Jul 2, 2009, at 1:55 PM, Paul Davis wrote:
>>>
>>>> On Thu, Jul 2, 2009 at 1:29 PM, Damien Katz<damien@apache.org> wrote:
>>>>>
>>>>> On Jul 2, 2009, at 1:16 PM, Jason Davies wrote:
>>>>>
>>>>>> On 2 Jul 2009, at 15:38, Brian Candler wrote:
>>>>>>
>>>>>>> For some fruit that was so low-hanging that I nearly stubbed
my toe on
>>>>>>> it,
>>>>>>> see https://issues.apache.org/jira/browse/COUCHDB-399
>>>>>>
>>>>>>
>>>>>> Nice work!  I'd be interested to see what kind of performance increase
>>>>>> we
>>>>>> get from Spidermonkey 1.8.1, which comes with native JSON
>>>>>> parsing/encoding.
>>>>>>  See here for details:
>>>>>> https://developer.mozilla.org/En/Using_native_JSON .
>>>>>>
>>>>>> Rumour has it 1.8.1 will be released any time soon (TM)
>>>>>
>>>>> I'm not sure the new engine is such a no-brainer. One thing about the
new
>>>>> generation of JS VMs is we've seen greatly increased memory usage with
>>>>> earlier versions. Also the startup times might be longer, or shorter.
>>>>>
>>>>> Though I wonder if this can be improved by forking a JS process rather
>>>>> than
>>>>> spawning a new process.
>>>>>
>>>>
>>>> Memory usage is a definite concern. I'm not sure I follow why startup
>>>> times would be important though. Am I missing something?
>>>
>>> Start up time isn't a huge concern, but it's is a something to consider. On
>>> a heavily loaded system, scripts that normally work might start to time out,
>>> requiring restarting the process. Lots of restarts may start to eat lots cpu
>>> and memory IO.
>>>
>>> -Damien
>>>
>>>
>>>>
>>>>> -Damien
>>>>>
>>>>>> --
>>>>>> Jason Davies
>>>>>>
>>>>>> www.jasondavies.com
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>

Mime
View raw message