Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Fri, 09 Oct 2015 11:38:19 -0700
From: Florin Andrei <florin@andrei.myip.org>
To: user@couchdb.apache.org
Subject: Re: DB only replies to 1k HTTP connections
Reply-To: user@couchdb.apache.org
Mail-Reply-To: user@couchdb.apache.org
In-Reply-To: 
 <CAKiq=tBw-d9NOHvThJ2XAWmRoqax3RvJ1AoJY9BMrMzqJM3AAQ@mail.gmail.com>
References: <9ce10aa8b3c23e41fdd6e35dd8eb3373@andrei.myip.org>
 <5617730D.8030005@extravision.com>
 <CAKiq=tBw-d9NOHvThJ2XAWmRoqax3RvJ1AoJY9BMrMzqJM3AAQ@mail.gmail.com>
Message-ID: <97e957251f21cc076393b5b8cff55b58@andrei.myip.org>
User-Agent: Roundcube Webmail/0.9.2

(replying to both Tom and Mike)

I suspect it's not running into resource exhaustion of the usual kind 
(CPU, etc), but I don't have a clear proof.

To begin with, it's hard to track CPU/disk/memory usage because the test 
is finished quickly; I am tracking _stats with Sensu and Graphite but I 
need to increase time resolution. Anyway, when I'm under the connection 
limit the system uses all CPU cycles and generates some decent disk 
usage but not too much. Userspace RAM is at less than 50%. The DB 
responds fairly quickly, within a dozen seconds for all requests, or so.

Some ~1k requests in the pool (the first ones that issue a GET) work 
just fine.
- node.js sends out "GET /db/_design/blah". CouchDB responds immediately 
with an empty ACK packet.
- 7 seconds later there's a PSH/ACK from CouchDB containing the HTTP 
response headers (response headers to GET)
- node.js responds with an ACK
- CouchDB respons immediately with a PSH/ACK containing the JSON 
response (small JSON document)
- immediately there's a FIN - FIN/ACK exchange both ways, closing the 
connection

The connections that are too late to send a GET are like this:
- node.js sends out "GET /db/_design/blah", and CouchDB responds with an 
empty ACK packet.
- And then there's silence on that connection.
- After 60 seconds, node.js times out its side of the channel by sending 
a FIN/ACK, to which the CouchDB instance responds with ACK.
- After another 60 seconds of silence, I get another ACK packet from 
CouchDB, containing the HTTP headers of the response to the GET, but no 
data. The node.js instance goes "what is this? I've closed this channel 
already" and responds with a RST packet.
The logic of the exchange is broken.

It's interesting that there's a cut off after the winning ~1k 
connections in the "contest". If it was CPU exhaustion, I would expect a 
gradual buildup of failures.

How many concurrent connections should I expect 1.6.1 to be able to 
handle? Surely there are people out there that are doing more than 1k 
concurrently. What's the cookbook recipe for that?

-- 
Florin Andrei
http://florin.myip.org/