From couchdb-user-return-1700-apmail-incubator-couchdb-user-archive=incubator.apache.org@incubator.apache.org Mon Nov 03 10:51:17 2008 Return-Path: Delivered-To: apmail-incubator-couchdb-user-archive@locus.apache.org Received: (qmail 4180 invoked from network); 3 Nov 2008 10:51:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 3 Nov 2008 10:51:17 -0000 Received: (qmail 99496 invoked by uid 500); 3 Nov 2008 10:51:22 -0000 Delivered-To: apmail-incubator-couchdb-user-archive@incubator.apache.org Received: (qmail 99456 invoked by uid 500); 3 Nov 2008 10:51:22 -0000 Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: couchdb-user@incubator.apache.org Delivered-To: mailing list couchdb-user@incubator.apache.org Received: (qmail 99445 invoked by uid 99); 3 Nov 2008 10:51:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2008 02:51:22 -0800 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.68.5.9] (HELO relay00.pair.com) (209.68.5.9) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 03 Nov 2008 10:50:05 +0000 Received: (qmail 11831 invoked from network); 3 Nov 2008 10:50:44 -0000 Received: from 96.33.90.152 (HELO ?192.168.1.200?) (96.33.90.152) by relay00.pair.com with SMTP; 3 Nov 2008 10:50:44 -0000 X-pair-Authenticated: 96.33.90.152 Message-Id: <42E2723A-38EF-4302-83D3-8959148AC3BF@apache.org> From: Damien Katz To: couchdb-user@incubator.apache.org In-Reply-To: <241FD008-0740-4393-9F4D-D2A7C14F764D@apache.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v929.2) Subject: Re: Largest CouchDB dbs? Date: Mon, 3 Nov 2008 05:50:44 -0500 References: <8C1EE3D1-D62F-4C6A-A859-196297E09C3D@apache.org> <46302F0F-7FF1-475B-B341-E7699295D915@apache.org> <241FD008-0740-4393-9F4D-D2A7C14F764D@apache.org> X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org On Nov 3, 2008, at 4:34 AM, Jan Lehnardt wrote: > > On Nov 3, 2008, at 10:26, Ask Bj=F8rn Hansen wrote: > >> >> On Nov 2, 2008, at 21:19, Paul Davis wrote: >> >>> View updates are admittedly slower than we'd like. There's planned >>> work on making this sort of thing parallelized to automagically fill >>> out multi-node clusters. Last I remember this is a 'probably 1.0' >>> feature though. >> >> >> I wonder if some of the slowness is from running it through JS? =20 >> Perl has an extremely fast JSON parser (JSON::XS), would there be =20 >> any potential optimization in using an alternate view language like =20= >> that - or is most of the time spent elsewhere? > > Another part is the Erlang to JSON conversion that is known to be =20 > slow-ish. I don't know > which side of the conversion weighs more though. I have to assume our Erlang json parser is much slower than the =20 spidermonkey one, which is just a call to eval(). Based on what I've =20 seen when profiling Erlang code, a built-in parser will give us a big =20= speed-up in view indexing. -Damien > We are working with the Erlang > community and developers to get a C-based JSON encoder into Erlang. =20= > Since building > a view server is relatively straightforward, giving a Perl version a =20= > shot is probably worth it. > > Which Erlang version are you running. R11B-x is known for causing =20 > view-build slowness. > R12B-3 and R12B-4 are good. > > >> I don't expect more than a few tens of thousands of new documents a =20= >> day though, so other than on index changes this isn't such a big =20 >> deal. > > That sounds about correct. > > Cheers > Jan > -- >