incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: couchdb on smp?
Date Mon, 04 May 2009 16:07:04 GMT
On Mon, May 4, 2009 at 11:58 AM, Chris Anderson <jchris@apache.org> wrote:
> On Mon, May 4, 2009 at 7:23 AM, Zachary Zolton <zachary.zolton@gmail.com> wrote:
>> @janl
>>
>> Perhaps he's asking why there's no activity on the other processor?
>>
>> I think his expectation here is that Map-Reduce would be parallelized.
>> Correct me if I'm wrong, but CouchDB does not yet exploit parallelism
>> in view indexing yet, right?
>>
>
> Correct. And the JavaScript view server doesn't make it easy to do so
> without starting multiple OS processes per map-function. I can see
> this being nifty down the road, but it's an optimization.
>
> However, interest is growing in an Erlang view server, and
> http://github.com/mmcdanie/erlview points out that the view process
> architecture could use some changes to make Erlang views parallel.
> Those changes will in turn make it more trivial to parallelize JS view
> computation, in the cases that need it. So I can see us getting to
> parallel JS views, but I think the cleaner way to get there is to
> start with proper Erlang views.
>

I'd also throw in that the biggest bottleneck for JS views is probably
the JSON serialization. I've been asked to try getting eep0018 in to a
copy of trunk as a configuration parameter. I'll get a branch on
github up by the end of the week to have a reference for that. My gut
feeling is that once we get that sorted out, we'll probably see that
disk I/O becomes the bottleneck. I could see maybe eeking out some
extra performance by streaming data through the map servers as opposed
to the serialized delegation method we're using now, but then it
becomes a cost/benfit of complexity. We'll see how it goes.

Paul Davis

>> —zdzolton
>>
>> On Mon, May 4, 2009 at 7:46 AM, Jan Lehnardt <jan@apache.org> wrote:
>>> Beam and couchjs pipe data back and  forth during view generation. While the
>>> one works, the other waits. The scheduler is smart enough to keep the
>>> processes local to a single CPU. Otherwise it's be even more expensive.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>> On 04.05.2009, at 12:50, Elf <elf2001@gmail.com> wrote:
>>>
>>>> Hello.
>>>> I'm using couchdb-0.9 (erlang 13.2) on my linux server with 4 CPUs
>>>> (Core 2 Quad).
>>>> Every time couchdb needs to reindex views, i see 2 serious processes
>>>> in htop - beam and couchjs. Each of them eats < 100% of 1 cpu, and sum
>>>> of their usage is about (but not greater) 100% of 1 cpu (80/20, 70/30
>>>> and so on).
>>>> Can somebody explain, why that 2 different processes (they have
>>>> differend PIDs) doesn't used different cpus - (one process for cpu)?
>>>>
>>>>
>>>> --
>>>> ----------------
>>>> Best regards
>>>> Elf
>>>> mailto:elf2001@gmail.com
>>>>
>>>
>>
>
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>

Mime
View raw message