Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=ohMAOftd9QYU1lwRUiN5JPy37q6+9GnVzdKuhT7hiS3XrxfBSY07kWGa/z35Zm2OpY5GQKDBEKrfx+tVDzqFOJVISUvbe6RBC0YRH8G6LUrziTXou/hmu10zcmVpGEG1udjHzhCknCcexc0FB8HsmF5b9PziM8vIZ6mHmvU+AeI=;
References: <1341538667.23741.YahooMailNeo@web39406.mail.mud.yahoo.com>
 <FE1F27F1-1E17-4B47-8235-37581DE9D5C8@gmail.com>
Message-ID: <1341613181.10647.YahooMailNeo@web39406.mail.mud.yahoo.com>
Date: Fri, 6 Jul 2012 15:19:41 -0700 (PDT)
From: Andreas Kemkes <a5sk4s@yahoo.com>
Reply-To: Andreas Kemkes <a5sk4s@yahoo.com>
Subject: Re: How many filtered replications is too many?
To: "user@couchdb.apache.org" <user@couchdb.apache.org>
In-Reply-To: <FE1F27F1-1E17-4B47-8235-37581DE9D5C8@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="411857043-933338849-1341613181=:10647"

--411857043-933338849-1341613181=:10647
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi Mathias:=0A=0AYes, you're analysis looks like it is spot on. =C2=A0There=
 are advantages though to use the replication feature - looking at the _cha=
nges feed, I'm not immediately clear on how I would achieve the same behavi=
or (e.g., deleted documents) - maybe due to my lack of exposure to couch de=
tails - even the _changes feed was new to me.=0A=0ABenoit's suggestion is a=
long the same lines and very interesting but requires an even newer couch i=
nstallation than I currently have in place.=0A=0AThat said, I also looked i=
nto if it is disk seek time that causes it, but iostat and its numbers for =
iowait suggest otherwise:=0A=0Aavg-cpu: =C2=A0%user =C2=A0 %nice %system %i=
owait =C2=A0%steal =C2=A0 %idle=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 90.36 =
=C2=A0 =C2=A00.00 =C2=A0 =C2=A03.71 =C2=A0 =C2=A00.10 =C2=A0 =C2=A05.19 =C2=
=A0 =C2=A00.64=0A=0ADevice: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0tps =
=C2=A0 =C2=A0kB_read/s =C2=A0 =C2=A0kB_wrtn/s =C2=A0 =C2=A0kB_read =C2=A0 =
=C2=A0kB_wrtn=0Axvdf =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 49.05 =C2=A0=
 =C2=A0 =C2=A0 799.60 =C2=A0 =C2=A0 =C2=A0 =C2=A031.57 =C2=A0 =C2=A0 =C2=A0=
 8004 =C2=A0 =C2=A0 =C2=A0 =C2=A0316=0A=0AI see many couchjs processes runn=
ing that require a constant CPU load of 6-9% each in top. Is that expected?=
=0A=0ATailing the log files for POSTs also makes me wonder about scheduling=
 fairness among the replications. =C2=A0I see activity mostly for a small n=
umber of the target databases. =C2=A0Do you know how this is being handled?=
=0A=0AThanks,=0A=0AAndreas=0A=0A=0A=0A________________________________=0A F=
rom: Mathias Leppich <mleppich@muhqu.de>=0ATo: user@couchdb.apache.org; And=
reas Kemkes <a5sk4s@yahoo.com> =0ASent: Friday, July 6, 2012 2:37 AM=0ASubj=
ect: Re: How many filtered replications is too many?=0A =0AHi Andreas,=0A=
=0AIf you say you want to split one large database in many smaller ones as =
a one-time task, its probably more efficient to write a script that reads t=
he _changes feed of the large database and then decides where to put each d=
ocument. Compared to the 200 filtered replications you will only need to re=
ad the changes feed 1 time instead of 200 times in parallel which will resu=
lt in very poor performance because of disk seek times=E2=80=A6=0A=0ASuch a=
 migration script is only a few lines of code. And the _changes feed also l=
ets you catchup after an initial split, you just need to log the passed seq=
 number to know where you left and start over.=0A=0A- mathias=0A=0AOn Jul 6=
, 2012, at 3:37 , Andreas Kemkes wrote:=0A=0A> I'm trying to split up a mon=
olithic database into smaller ones using filtered continuous replications i=
n couchdb 1.2.=0A> =0A> I need about 200 of these replications (on a single=
 server) and would like to parallelize as much as possible.=C2=A0 Yet, when=
 I do, the cpu load gets very high and the system seems to be crawling, rep=
lication seems to be slow, and I'm seeing timeout and other errors.=0A> =0A=
> How can I best determine what the bottleneck is?=0A> =0A> Are there sugge=
stions on how to configure couchdb to handle it better (I've increased max_=
dbs_open to 200)?=0A> =0A> How do I best achieve good throughput?=0A> =0A> =
This will be a one-time task, so any large measurement / monitoring effort =
is probably overkill.=0A> =0A> Any suggestions are much appreciated (includ=
ing suggestions for different approaches).=0A> =0A> Thanks,=0A> =0A> Andrea=
s
--411857043-933338849-1341613181=:10647--