Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Apple Message framework v1283)
Subject: Re: tinkering with limits while replicating
From: Nathan Vander Wilt <nate-lists@calftrail.com>
In-Reply-To: <097C0DAA-FF04-4EC7-905C-22310DA33EC1@gmail.com>
Date: Tue, 5 Feb 2013 10:22:44 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <AA43831B-3937-4C7E-B718-30C8296D5C75@calftrail.com>
References: <097C0DAA-FF04-4EC7-905C-22310DA33EC1@gmail.com>
To: user@couchdb.apache.org

Hi Stephen,

I've been doing some tests related to replication lately too =
(continuous+filtered in my case). I suspect the reason Futon hangs is =
because your whole VM is running out of RAM due to your very high =
os_process_limit. I went in to a bit more detail in =
http://mail-archives.apache.org/mod_mbox/couchdb-dev/201302.mbox/%3c70278F=
4A-FD08-4818-89B7-EA1B0AF846F5@gmail.com%3e but this setting basically =
determines the size of the couchjs worker pool =97 you'd probably rather =
have a bit of contention for the pool at a reasonable size (maybe ~100 =
per GB free, tops?) than start paging.

hth,
-natevw


On Feb 4, 2013, at 5:15 PM, Stephen Bartell wrote:

> Hi all,
>=20
> I'm hitting some limits while replicating , I'm hoping someone could =
advise. =20
> Im running this in a VM on my macbook with the following allocated =
resources:
> ubuntu 11.04
> 4 cores @ 2.3ghz
> 8 gb mem
>=20
> I'm doing a one-to-many replication. =20
> 1) I create one db named test.=20
> 2) Then create [test_0 .. test_99] databases. =20
> 3) I then set up replications from test -> [test_0 .. test_99].  100 =
replications total.
> 4) I finally go to test and create a doc, hit save.
>=20
> When I hit save, futon becomes completely unresponsive for around =
10sec.  It eventually returns to normal behavior.
>=20
> Tailing the couchdb log I find waves of the following errors:
> [Tue, 05 Feb 2013 00:46:26 GMT] [info] [<0.6936.1>] Retrying POST =
request to http://admin:*****@localhost:5984/test_25/_revs_diff in 1.0 =
seconds due to error {code,503}
>=20
> I see that the replicator is finding the server to be unresponsive.  =
The waves of these messages show that replicator retries in 0.25 sec, =
then 0.5 sec, then 1sec, then 2sec.  This is expected.  Everything =
settles done after about 4 retries. =20
>=20
> So my first thought is resource limits.  I threw the book at it and =
set :
> 1) max_dbs_open: 500
> 2) os_process_limit: 5000
> 3) http_connections: 20000
> 4) ulimit -Sn 4096 (the hard limit is 4096)
>=20
> I really don't know whats reasonable for these values relative to how =
many replications I am setting up.  So these values, save max_dbs_open,  =
are all stabs in the dark.
>=20
> No change in performance.
>=20
> So, I'm at a loss now.  what can I do to get all this to work? Or what =
am I doing wrong?  And note that this is only a test.  I aim to =
quadruple  the amount of replications and have lots and lots of =
insertions on the so called "test" database.  Actually, there will be =
several of these one-to-many databases.
>=20
> I've heard people get systems up to thousands of dbs and replicators =
running just fine.  So I hope Im just not offering to right sacrifices =
up to couchdb yet.
>=20
> Thanks for any insight,
>=20
> sb
>=20