couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cameron Jacobson <came...@datashovel.com>
Subject concurrency in views
Date Tue, 06 Aug 2013 15:50:24 GMT
Hi,

I apologize if this has already reached a list.  I tried sending
previously, but based on the online archives it doesn't appear that it
was received by the list.


Original Message:
-----------------------------------

I've got a question regarding how CouchDB handles concurrency in views.

Background:  I'm writing a CouchDB client where I'm trying to leverage
libevent so the client can handle requests concurrently.

When I started to benchmark the "async" functionality I noticed that no
matter what I did the time-spent in the closure where all the buffers
were being retrieved was growing linearly.  My expectation, given that
this is concurrent accesses, would be that time would at worst grow
logarithmically (at least while the server had enough resources to
handle the requests concurrently).  No noticeable issues in other
critical areas, such as I/O traffic thresholds, CPU or memory bottlenecks.

I made several adjustments in the default.ini to ensure there were
enough resources allocated to the server.  I tried the "stale"
parameter, and "warmed" the views with pre-test queries.  Thinking it
was possible the view handling was the bottleneck I broke out my queries
against views1-10, all doing exactly the same thing.  Nothing seemed to
help.  This got me thinking that I may be implementing the libevent code
incorrectly.  I realized this wasn't the issue once I ran the code
against a remote environment (non-couchdb) and was getting no noticeable
increase in time over as many as 100 concurrent requests (tests on
couchdb until now have not exceeded 50 concurrent requests).

I don't have the background in CouchDB to know that it's the bottleneck,
or how I would go about proving this, but based on all the tests so far
I think I've narrowed in on CouchDB as the cause.  I'm running version
1.3.0.

I came across the following link, which may explain part of the problem,
but I don't think it explains why multiple concurrent queries across
multiple views showed the same performance characteristics as multiple
concurrent queries against a single view.  Also the comments in the
following thread are talking about large documents / views.  My tests
were against 10 "cloned" views which all only contained 2 documents,
each of which was less than 1kb in size:
http://comments.gmane.org/gmane.comp.db.couchdb.user/22200

Any insights into how I can tweak my configuration or what measures I
can take to improve this would be greatly appreciated.

Thanks

Mime
View raw message