Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Date: Tue, 6 Aug 2013 16:29:15 -0400
From: Joan Touzet <wohali@apache.org>
To: user@couchdb.apache.org, cameron@datashovel.com
Subject: Re: concurrency in views
Message-ID: <20130806202915.GA1913@atypical.net>
References: <52011B40.2020309@datashovel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <52011B40.2020309@datashovel.com>
User-Agent: Mutt/1.5.21 (2010-09-15)

Hi Cameron,

When you split up your views, did you split them into separate view
groups (different design docs) or keep them in the same view group
(ddoc)? If not, try to do so. This should lead to additional couchjs
processes being spawned for each view group, which (if your server can
support it) will allow increased parallelism.

That said, given your data set, I would be inclined to implicate your
libevent code. If you can attempt to reproduce the situation using
something "lower tech" such as multiple shell scripts + curl, this might
be more useful information.

On Tue, Aug 06, 2013 at 11:50:24AM -0400, Cameron Jacobson wrote:
> Hi,
> 
> I apologize if this has already reached a list.  I tried sending
> previously, but based on the online archives it doesn't appear that it
> was received by the list.
> 
> 
> Original Message:
> -----------------------------------
> 
> I've got a question regarding how CouchDB handles concurrency in views.
> 
> Background:  I'm writing a CouchDB client where I'm trying to leverage
> libevent so the client can handle requests concurrently.
> 
> When I started to benchmark the "async" functionality I noticed that no
> matter what I did the time-spent in the closure where all the buffers
> were being retrieved was growing linearly.  My expectation, given that
> this is concurrent accesses, would be that time would at worst grow
> logarithmically (at least while the server had enough resources to
> handle the requests concurrently).  No noticeable issues in other
> critical areas, such as I/O traffic thresholds, CPU or memory bottlenecks.
> 
> I made several adjustments in the default.ini to ensure there were
> enough resources allocated to the server.  I tried the "stale"
> parameter, and "warmed" the views with pre-test queries.  Thinking it
> was possible the view handling was the bottleneck I broke out my queries
> against views1-10, all doing exactly the same thing.  Nothing seemed to
> help.  This got me thinking that I may be implementing the libevent code
> incorrectly.  I realized this wasn't the issue once I ran the code
> against a remote environment (non-couchdb) and was getting no noticeable
> increase in time over as many as 100 concurrent requests (tests on
> couchdb until now have not exceeded 50 concurrent requests).
> 
> I don't have the background in CouchDB to know that it's the bottleneck,
> or how I would go about proving this, but based on all the tests so far
> I think I've narrowed in on CouchDB as the cause.  I'm running version
> 1.3.0.
> 
> I came across the following link, which may explain part of the problem,
> but I don't think it explains why multiple concurrent queries across
> multiple views showed the same performance characteristics as multiple
> concurrent queries against a single view.  Also the comments in the
> following thread are talking about large documents / views.  My tests
> were against 10 "cloned" views which all only contained 2 documents,
> each of which was less than 1kb in size:
> http://comments.gmane.org/gmane.comp.db.couchdb.user/22200
> 
> Any insights into how I can tweak my configuration or what measures I
> can take to improve this would be greatly appreciated.
> 
> Thanks

-- 
Joan Touzet | joant@atypical.net | wohali everywhere else