couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <rnew...@apache.org>
Subject Re: Post-mortem
Date Fri, 11 May 2012 12:44:47 GMT
Fundamentally, the issue is that updating a view is processing an
incoming, ordered list of changes, there's not much parallelism to be
had there. The reason sharding works is that you then have two smaller
lists which can be processed in parallel (but each still processed
serially). And, yes, my thought was that BigCouch will allow multiple
shards of the database on a single node, allowing parallel view builds
within that node.

Some care must be taken if we pursue optimizing a single view build.
In the context of a production system, with more than one database and
more than one view, all the cores get plenty of exercise. If every
task could run on all cores then I think you might hit other issues.

B.

On 11 May 2012 13:35, Dirkjan Ochtman <dirkjan@ochtman.nl> wrote:
> On Fri, May 11, 2012 at 2:29 PM, Robert Newson <rnewson@apache.org> wrote:
>> Making a single view indexing process faster keeps coming up. For one
>> thing, it's not that easy, otherwise it would have been done by now.
>> For another, this problem vanishes when you shard (and the BigCouch
>> merge will bring this to CouchDB).
>
> What does sharding mean in this context? Running CouchDB/BigCouch on
> multiple servers, or just running multiple processes on a single box?
> If the latter, why can't we run multiple threads/Erlang processes
> within a single shard/OS process? If the former, that's kind of silly,
> in the sense that building indexes (at least for me) is CPU-bound but
> leaves many of the cores in my server idle.
>
> Cheers,
>
> Dirkjan

Mime
View raw message