couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: using mutliple cores for view computation
Date Fri, 20 Mar 2009 11:54:09 GMT
On Mar 20, 2009, at 7:35 AM, Anand Chitipothu wrote:

>> I don't see anything out of the ordinary there. If you're not on  
>> trunk
>> then you'll definitely want to upgrade.
>
> my working copy is just 3 days old.
>
> Revision: 755487
> Last Changed Date: 2009-03-17 11:52:57 +0000 (Tue, 17 Mar 2009)
>
> I have about 30M documents and size of database is 26G.
>
> One thing that is worrying me is couchjs is not taking much cpu, most
> of the time cpu is taken by beam.smp. And it looks like both of them
> are sharing single core. At least, these 2 processes can take one core
> each. Any idea why it is not so?

Hi Anand, this matches my experience.  I believe the reason they  
appear to share a core is that the Erlang process waits on couchjs to  
send back results before continuing on with the indexing.  The  
workflow looks like

1) Pull docs into memory until we reach some threshold

2) map over docs:
     - convert doc to JSON
     - send JSON to view server
     - receive result from view server

3) Update the btrees with the results from the view server

4) Repeat

As others have said, serious attempts to optimize this workflow  
haven't really begun yet.  Best, Adam

Mime
View raw message