Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of snbartell@gmail.com
 designates 209.85.220.47 as permitted sender)
From: Stephen Bartell <snbartell@gmail.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_99B233B7-D453-42BC-9A76-A284DC2D738F"
Subject: tinkering with limits while replicating
Message-Id: <097C0DAA-FF04-4EC7-905C-22310DA33EC1@gmail.com>
Date: Mon, 4 Feb 2013 17:15:44 -0800
To: "user@couchdb.apache.org" <user@couchdb.apache.org>
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))

--Apple-Mail=_99B233B7-D453-42BC-9A76-A284DC2D738F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Hi all,

I'm hitting some limits while replicating , I'm hoping someone could =
advise. =20
Im running this in a VM on my macbook with the following allocated =
resources:
ubuntu 11.04
4 cores @ 2.3ghz
8 gb mem

I'm doing a one-to-many replication. =20
1) I create one db named test.=20
2) Then create [test_0 .. test_99] databases. =20
3) I then set up replications from test -> [test_0 .. test_99].  100 =
replications total.
4) I finally go to test and create a doc, hit save.

When I hit save, futon becomes completely unresponsive for around 10sec. =
 It eventually returns to normal behavior.

Tailing the couchdb log I find waves of the following errors:
[Tue, 05 Feb 2013 00:46:26 GMT] [info] [<0.6936.1>] Retrying POST =
request to http://admin:*****@localhost:5984/test_25/_revs_diff in 1.0 =
seconds due to error {code,503}

I see that the replicator is finding the server to be unresponsive.  The =
waves of these messages show that replicator retries in 0.25 sec, then =
0.5 sec, then 1sec, then 2sec.  This is expected.  Everything settles =
done after about 4 retries. =20

So my first thought is resource limits.  I threw the book at it and set =
:
1) max_dbs_open: 500
2) os_process_limit: 5000
3) http_connections: 20000
4) ulimit -Sn 4096 (the hard limit is 4096)

I really don't know whats reasonable for these values relative to how =
many replications I am setting up.  So these values, save max_dbs_open,  =
are all stabs in the dark.

No change in performance.

So, I'm at a loss now.  what can I do to get all this to work? Or what =
am I doing wrong?  And note that this is only a test.  I aim to =
quadruple  the amount of replications and have lots and lots of =
insertions on the so called "test" database.  Actually, there will be =
several of these one-to-many databases.

I've heard people get systems up to thousands of dbs and replicators =
running just fine.  So I hope Im just not offering to right sacrifices =
up to couchdb yet.

Thanks for any insight,

sb


--Apple-Mail=_99B233B7-D453-42BC-9A76-A284DC2D738F--