Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of norman.barker@gmail.com
 designates 209.85.210.204 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=IpY/yKeGhQHvyKMdwRfYuI/QM+VOU/AKQq4HCHvL0XBs1YTirS8bomW1XIlIQz2On3
         I4Mqzrhueg3F31Av6+pukSZMQARrAuYRQ7IKdmWjsmf0PHdUBm8QK70CewTIxdByQ7lg
         q0PGAPTN6dZULCdJ5oKVZCpTjmWAomqHo44/I=
MIME-Version: 1.0
In-Reply-To: <921000908091913t68af7c3k3dcd8b60d9368d8c@mail.gmail.com>
References: <921000908082201k5caa1cf6pf00283012f46457b@mail.gmail.com>
	 <e2111bbb0908082206i17e80d49n80bc0e806522a1bc@mail.gmail.com>
	 <4A7E6542.10408@borwankar.com>
	 <5A3FA940-B720-4B85-827F-CBC9142762C3@apache.org>
	 <4A7EFBC4.2060402@borwankar.com>
	 <31CC10A5-3C7D-4B7C-B65E-DCAFBD287246@apache.org>
	 <921000908091425l6af7f04dvf7e486af1b4cfc7e@mail.gmail.com>
	 <e282921e0908091442n31ee6f9cxd2a4def676e9fcff@mail.gmail.com>
	 <921000908091913t68af7c3k3dcd8b60d9368d8c@mail.gmail.com>
Date: Mon, 10 Aug 2009 07:53:35 -0600
Message-ID: <fbbaf7b0908100653p4b258062ga25544de807175f0@mail.gmail.com>
Subject: Re: couchdb on 8 core redhat/fedora
From: Norman Barker <norman.barker@gmail.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I was a little surprised by this, and did a quick grep on spawn on the
couchdb code base, it isn't used that much and I wonder why. In
particular for bulk doc updates I wonder why pmap isn't used (perhaps
I am missing something), for my project I have needed to write a
custom updater, a pre-writer, (great modular design now btw in 0.9 I
was able to hook it all up - updater, query server from the ini file)
and I am going up to a javascript function to parse an incoming
document (not JSON) into multiple JSON documents I am then using a
pmap to write this to the db, doesn't seem to be that complicated and
I should then get an increase in speed to compensate for the initial
javascript parsing delay.

Could all (or most) use of the plain map be replaced with a pmap in
couchdb or am I missing something?

Not to hijack this thread, but I have written a pre-writer, are there
any plans to add filters (pre and post) on the couchdb roadmap?

As a J2EE developer, I still see the benefit of the Erlang VM - easier
to develop 'fast' applications in for sure.

thanks,

Norman

On Sun, Aug 9, 2009 at 8:13 PM, Nitin Borwankar<nitin@borwankar.com> wrote:
> I believe there are at least 4 spindles. =A0But I don't expect to be
> continually writing or writing while generating indexes - this is meant t=
o
> be primarily a reference "datawarehouse" kind of read-only database - wri=
tes
> will happen in batch mode once a day for a couple of hours during which t=
ime
> no reads will be taking place. =A0All initial view generation will happen=
 when
> no writes are taking place.
> Having views on separate disk from db is a good suggestion =A0I'll see ab=
out
> the symlinks part as well.
>
> Thanks much all for all the info - this has been extremely useful and
> helpful
>
> Nitin
>
> 37% of all statistics are made up on the spot
> -------------------------------------------------------------------------=
------------
> Nitin Borwankar
> nborwankar@gmail.com
>
>
> On Sun, Aug 9, 2009 at 2:42 PM, Chris Anderson <jchris@apache.org> wrote:
>
>> On Sun, Aug 9, 2009 at 2:25 PM, Nitin Borwankar<nitin@borwankar.com>
>> wrote:
>> > On Sun, Aug 9, 2009 at 12:13 PM, Adam Kocoloski <kocolosk@apache.org>
>> wrote:
>> >
>> >>
>> >> Hi Nitin, Jan's right, if you're only building views from a single
>> design
>> >> doc you won't get much indexing speedup from multi-core at the moment=
.
>> =A0We
>> >> do spawn multiple couchjs processes (often one does the map and the
>> other
>> >> the reduce), but we don't map docs out to them simultaneously or
>> anything.
>> >> =A0Also, the Erlang process communicating with couchjs blocks and wai=
ts
>> for
>> >> the results when it sends data out. =A0Best,
>> >>
>> >> Adam
>> >>
>> >
>> > Ok, so I could have multiple databases each with multiple design docs =
and
>> > make a number of requests and have a number of couchjs processes sprea=
d
>> out
>> > across multiple cores right?
>> >
>> > I suppose what I am hearing is that views in a single design doc can
>> become
>> > a bottleneck but views in multiple design docs will get the benefit of
>> > multiple couchjs processes.
>> >
>> > Since this part of my experiment is not couchapp based, I am completel=
y
>> fine
>> > having a) multiple databases and b) multiple design docs per database.
>> >
>>
>> Unless you have lots of disks, you won't gain performance from
>> multiple dbs. However, you might do well you put views onto a
>> different disk from the db file, if you'll be writing while generating
>> indexes. or spread view files across disks with symlinks so disks
>> don't have to seek as much to record view rows.
>>
>> > So this gives me the OS level scheduling of couchjs processes across
>> cores,
>> > I think.
>> >
>> > Nitin
>> >
>> >
>> >
>> > 37% of all statistics are made up on the spot
>> >
>> ------------------------------------------------------------------------=
-------------
>> > Nitin Borwankar
>> > nborwankar@gmail.com
>> >
>>
>>
>>
>> --
>> Chris Anderson
>> http://jchrisa.net
>> http://couch.io
>>
>