couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cameron Jacobson <came...@datashovel.com>
Subject Re: concurrency in views
Date Tue, 06 Aug 2013 21:07:56 GMT
Joan,

Thanks for the advice.  In some initial tests across only 3 design docs
I'm seeing a noticeable speed increase.  Granted with the workload I'm
talking about we're dealing with times measured in fractions of
milliseconds, but it's enough of an improvement to know it's a proper
way to take advantage of the concurrency / parallelism on the server
side as well.

Thank you,
Cameron


On 08/06/2013 04:29 PM, Joan Touzet wrote:
> Hi Cameron,
>
> When you split up your views, did you split them into separate view
> groups (different design docs) or keep them in the same view group
> (ddoc)? If not, try to do so. This should lead to additional couchjs
> processes being spawned for each view group, which (if your server can
> support it) will allow increased parallelism.
>
> That said, given your data set, I would be inclined to implicate your
> libevent code. If you can attempt to reproduce the situation using
> something "lower tech" such as multiple shell scripts + curl, this might
> be more useful information.
>
> On Tue, Aug 06, 2013 at 11:50:24AM -0400, Cameron Jacobson wrote:
>> Hi,
>>
>> I apologize if this has already reached a list.  I tried sending
>> previously, but based on the online archives it doesn't appear that it
>> was received by the list.
>>
>>
>> Original Message:
>> -----------------------------------
>>
>> I've got a question regarding how CouchDB handles concurrency in views.
>>
>> Background:  I'm writing a CouchDB client where I'm trying to leverage
>> libevent so the client can handle requests concurrently.
>>
>> When I started to benchmark the "async" functionality I noticed that no
>> matter what I did the time-spent in the closure where all the buffers
>> were being retrieved was growing linearly.  My expectation, given that
>> this is concurrent accesses, would be that time would at worst grow
>> logarithmically (at least while the server had enough resources to
>> handle the requests concurrently).  No noticeable issues in other
>> critical areas, such as I/O traffic thresholds, CPU or memory bottlenecks.
>>
>> I made several adjustments in the default.ini to ensure there were
>> enough resources allocated to the server.  I tried the "stale"
>> parameter, and "warmed" the views with pre-test queries.  Thinking it
>> was possible the view handling was the bottleneck I broke out my queries
>> against views1-10, all doing exactly the same thing.  Nothing seemed to
>> help.  This got me thinking that I may be implementing the libevent code
>> incorrectly.  I realized this wasn't the issue once I ran the code
>> against a remote environment (non-couchdb) and was getting no noticeable
>> increase in time over as many as 100 concurrent requests (tests on
>> couchdb until now have not exceeded 50 concurrent requests).
>>
>> I don't have the background in CouchDB to know that it's the bottleneck,
>> or how I would go about proving this, but based on all the tests so far
>> I think I've narrowed in on CouchDB as the cause.  I'm running version
>> 1.3.0.
>>
>> I came across the following link, which may explain part of the problem,
>> but I don't think it explains why multiple concurrent queries across
>> multiple views showed the same performance characteristics as multiple
>> concurrent queries against a single view.  Also the comments in the
>> following thread are talking about large documents / views.  My tests
>> were against 10 "cloned" views which all only contained 2 documents,
>> each of which was less than 1kb in size:
>> http://comments.gmane.org/gmane.comp.db.couchdb.user/22200
>>
>> Any insights into how I can tweak my configuration or what measures I
>> can take to improve this would be greatly appreciated.
>>
>> Thanks


Mime
View raw message