From dev-return-4915-apmail-couchdb-dev-archive=couchdb.apache.org@couchdb.apache.org Sat Jul 04 18:40:03 2009 Return-Path: Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: (qmail 95142 invoked from network); 4 Jul 2009 18:40:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 4 Jul 2009 18:40:03 -0000 Received: (qmail 16758 invoked by uid 500); 4 Jul 2009 18:40:12 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 16669 invoked by uid 500); 4 Jul 2009 18:40:12 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 16659 invoked by uid 99); 4 Jul 2009 18:40:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 18:40:12 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of sshumaker@gmail.com designates 209.85.221.191 as permitted sender) Received: from [209.85.221.191] (HELO mail-qy0-f191.google.com) (209.85.221.191) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 18:40:01 +0000 Received: by qyk29 with SMTP id 29so979363qyk.13 for ; Sat, 04 Jul 2009 11:39:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=TYH6hM1qxCLgrAc3H1wfb+yOvXOHX9I+SMZBikQMvzg=; b=IeKNz9xD7h+D6y6Uv4sMQlDMOVjG2Xz8DYy4iXZyffRlHHvV8fi9+Dkj8ARSGKxdYF B2beqOkkB2KlhyI+lGxw/YWJdy40CRkizRg2jGGR4s2COSunwX9bguK7r4T8HbHCYmBb SxW+7DR6QpGetRSkNxtjjPYPToOSvRuwXMu8g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=h0WhT4ISwwSsZ+KsIXa1KgVWhkCxgqWA9Q9hkKBpuAs40SUTAOJuCsrm/zOvpKieHv UjH6dkqSPLGP0OAa4WnJ7Hgrxi8E4v4Ggp1sYmFGOqQlXx/OZoL07nNKUMtko7zqhCv4 F1+oG8eD1tdwKNu7gGvbVJ49k9iWJ3Xid1b1c= MIME-Version: 1.0 Received: by 10.224.89.1 with SMTP id c1mr3307079qam.95.1246732779536; Sat, 04 Jul 2009 11:39:39 -0700 (PDT) In-Reply-To: <261cf6280907021617r7c84c91ak71605ee619556000@mail.gmail.com> References: <060429AB-10DE-482B-8993-27FBD486F191@jasondavies.com> <4C82520A-A9E1-4F84-8630-0ED3810D8E61@apache.org> <77950CB1-716C-4C5D-B97E-3C398F728E15@apache.org> <261cf6280907021415x3746a14cm4cbe0cf914a951cf@mail.gmail.com> <261cf6280907021442w48882a1fqd563fdf4af39486d@mail.gmail.com> <261cf6280907021450x48c8d5beob8b8c79e4091b31f@mail.gmail.com> <261cf6280907021617r7c84c91ak71605ee619556000@mail.gmail.com> Date: Sat, 4 Jul 2009 11:39:39 -0700 Message-ID: <261cf6280907041139p73f673eehe422f9df9abe88b6@mail.gmail.com> Subject: Re: View Performance (was Re: The 1.0 Thread) From: Scott Shumaker To: dev@couchdb.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Compiling with HiPE didn't seem to make any difference in performance. :( On Thu, Jul 2, 2009 at 4:17 PM, Scott Shumaker wrote: > I'll try that out tomorrow and post the results here. > > On Thu, Jul 2, 2009 at 3:01 PM, Paul Davis w= rote: >> On Thu, Jul 2, 2009 at 5:50 PM, Scott Shumaker wrot= e: >>> One question, though: Why are the emitted view results stored as >>> erlang terms, as opposed to storing the JSON returned from the view >>> server - which is what you'll be serving to the clients anyway? >>> >>> If you skipped the reverse json->erlang encoding, and additionally >>> stored a cached json copy of each document alongside the document >>> whenever a document in couchdb was created/updated (which you could >>> incrementally generate in a separate erlang process so you don't have >>> to slow down write performance) - and just pass this json copy to the >>> view, you could basically eliminate the json->erlang conversion >>> overhead entirely (since it would only be done asynchronously). >>> >>> Even if you need to store the emitted view results back into erlang, >>> you could have a special optimization case for emitting (key, doc) - >>> because you already have the document as both erlang/json (assuming >>> you were storing cached json copies). =A0And include_docs would get >>> faster since you wouldn't need to do the json conversion there either. >>> >>> Just a thought. >>> >> >> Premature optimization is the root of all evil? Have you tried >> compiling CouchDB with HiPE enabled. I'm inclined to agree with you >> that the large JSON values are probably a significant cause here. >> Assuming your Erlang is HiPE enabled you can do something like this to >> compile CouchDB: >> >> =A0 =A0$ ./bootstrap >> =A0 =A0$ ERLC_FLAGS=3D"+native +inline +inline_list_funcs" ./configure >> =A0 =A0$ make >> =A0 =A0$ sudo make install >> >> >>> Scott >>> >>> On Thu, Jul 2, 2009 at 2:42 PM, Scott Shumaker wro= te: >>>> I should mention that we tend to emit (doc._id, doc) in our views - as >>>> opposed to doc._id, null and using include_docs - because we found >>>> that doc._id,null gave us a 30% speedup on building the views, but >>>> cost us about the same on each additional hit to the view. >>>> >>>> Scott >>>> >>>> On Thu, Jul 2, 2009 at 2:15 PM, Scott Shumaker wr= ote: >>>>> We see times that are considerably worse. =A0We mostly have maps - ve= ry >>>>> few reduces. =A0We have 40k objects, about 25 design docs, and 90 vie= ws. >>>>> =A0Although we're about to change the code to auto-generate the desig= n >>>>> docs based on the view filters used (re: view filter patch) - see if >>>>> that helps. >>>>> >>>>> Maybe it's because we have larger objects - but re-indexing a typical >>>>> new view takes > 5 minutes (with view filtering off). =A0Some are wor= se. >>>>> =A0With view filtering on some can be quite fast - some views finish = in >>>>> like 10 seconds. =A0Interestingly, reindexing all views takes about a= n >>>>> hour - with or without view filtering. =A0I'm guessing that a >>>>> substantial part of the bottleneck is erlang -> json serialization. >>>>> Many of our objects are heavily nested structures and exceed 10k in >>>>> size. =A0One other note - when we tried dropping in the optimized >>>>> 'main.js' posted on the mailing list, we saw an overall 20% speedup. >>>>> Unfortunately, it wasn't compatible with the authentication stuff, an= d >>>>> the deployment was a bit wacky, so we're holding off on that right >>>>> now. >>>>> >>>>> >>>>> On Thu, Jul 2, 2009 at 11:30 AM, Damien Katz wrote= : >>>>>> >>>>>> On Jul 2, 2009, at 1:55 PM, Paul Davis wrote: >>>>>> >>>>>>> On Thu, Jul 2, 2009 at 1:29 PM, Damien Katz wrot= e: >>>>>>>> >>>>>>>> On Jul 2, 2009, at 1:16 PM, Jason Davies wrote: >>>>>>>> >>>>>>>>> On 2 Jul 2009, at 15:38, Brian Candler wrote: >>>>>>>>> >>>>>>>>>> For some fruit that was so low-hanging that I nearly stubbed my = toe on >>>>>>>>>> it, >>>>>>>>>> see https://issues.apache.org/jira/browse/COUCHDB-399 >>>>>>>>> >>>>>>>>> >>>>>>>>> Nice work! =A0I'd be interested to see what kind of performance i= ncrease >>>>>>>>> we >>>>>>>>> get from Spidermonkey 1.8.1, which comes with native JSON >>>>>>>>> parsing/encoding. >>>>>>>>> =A0See here for details: >>>>>>>>> https://developer.mozilla.org/En/Using_native_JSON=A0. >>>>>>>>> >>>>>>>>> Rumour has it 1.8.1 will be released any time soon (TM) >>>>>>>> >>>>>>>> I'm not sure the new engine is such a no-brainer. One thing about = the new >>>>>>>> generation of JS VMs is we've seen greatly increased memory usage = with >>>>>>>> earlier versions. Also the startup times might be longer, or short= er. >>>>>>>> >>>>>>>> Though I wonder if this can be improved by forking a JS process ra= ther >>>>>>>> than >>>>>>>> spawning a new process. >>>>>>>> >>>>>>> >>>>>>> Memory usage is a definite concern. I'm not sure I follow why start= up >>>>>>> times would be important though. Am I missing something? >>>>>> >>>>>> Start up time isn't a huge concern, but it's is a something to consi= der. On >>>>>> a heavily loaded system, scripts that normally work might start to t= ime out, >>>>>> requiring restarting the process. Lots of restarts may start to eat = lots cpu >>>>>> and memory IO. >>>>>> >>>>>> -Damien >>>>>> >>>>>> >>>>>>> >>>>>>>> -Damien >>>>>>>> >>>>>>>>> -- >>>>>>>>> Jason Davies >>>>>>>>> >>>>>>>>> www.jasondavies.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >