incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McDaniel <couc...@autosys.us>
Subject Re: Chance of including CouchDB in Linux distros or desktops?
Date Fri, 06 Mar 2009 07:44:04 GMT
On Thu, Mar 05, 2009 at 08:38:11PM -0800, Chris Anderson wrote:
> On Thu, Mar 5, 2009 at 8:17 PM, kowsik <kowsik@gmail.com> wrote:
> > Speaking of couchlite, I would __love__ to see parallel map/reduce. I
> > got this 100K+ doc database and the instant-gratification-factor for
> > trying out new views just sucks (takes about 15 minutes). I have to
> > reduce the doc count, just so I can try out new ideas in terms of
> > views. And it's on a dual quad-core system. The other 7 cpus badly
> > want to relax on the couch, but can't. :-(
> 
> I haven't had a chance yet, but one science experiment that might help
> with identifying bottlenecks would be Michael McDaniel's erlang view
> server:
> http://github.com/mmcdanie/erlview/tree/master
> 
> My gut tells me that the bottleneck is index insertion, and not view
> function execution. Erlang views should have almost no overhead, so we
> can guess at the best case for parallel JS view servers. It maybe that
> to get true parallelism we need to parallelize indexing and not just
> generation...
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 I agree with paralleizing indexing, Chris.  Also, being able to 
 have different index/views in different files could speed things
 up, especially when every map fun has to run again; e.g. place
 the different index/view files on different spindles.

 Regarding parallizing views, that is TODO #7 in erlview.  If
 couch_query_servers:map_docs/2 sent a list of documents, the
 erlview:handle_call( {map_doc ... ) could split up the list
 and run parallel funs.  couch_query_servers could dynamically
 determine how many docs to send in the list by how much memory
 is currently available.


 Do note that early erlview testing has shown it to be two to 
 ten times faster than javascript views, depending on the 
 map fun complexity.  I attribute this greatly to not having
 to do json <-> Erlang term() conversions.


~Michael


> 
> 
> -- 
> Chris Anderson
> http://jchris.mfdz.com

-- 
Michael McDaniel
Portland, Oregon, USA
http://trip.autosys.us
http://autosys.us



Mime
View raw message