Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
From: Joey Samonte <csharpdeveloper@hotmail.com>
To: "dev@couchdb.apache.org" <dev@couchdb.apache.org>, Joan Touzet
	<wohali@apache.org>
Subject: Re: High CPU usage caused by clustering and/or replication?
Thread-Topic: High CPU usage caused by clustering and/or replication?
Thread-Index: AQHSDYgcPrswn7modEmrLcCn02LcZKBxkY4AgAa5dG6Tnk4fsOxhwqeLPTxXrBP+Fo+ctQ==
Date: Wed, 14 Sep 2016 10:59:12 +0000
Message-ID: <BN4PR10MB0995D26B5AF047D8386F2A0CA2F10@BN4PR10MB0995.namprd10.prod.outlook.com>
References: <BAY174-W27E4B81B76BF4D2132E13BA2FA0@phx.gbl>
 <10492210.359.1473419538926.JavaMail.Joan@RITA>
 <BAY174-W25B0C3C1ABF4BAD99B8937A2FA0@phx.gbl>
 <2883838.505.1473451976224.JavaMail.Joan@RITA>
 <BN4PR10MB0995EC1FA0BF1E268EBD2C2BA2F10@BN4PR10MB0995.namprd10.prod.outlook.com>
 <25200628.1118.1473824952732.JavaMail.Joan@RITA>
 <BN4PR10MB0995D1EA62B6732AD0008E05A2F10@BN4PR10MB0995.namprd10.prod.outlook.com>,<27966981.1153.1473829264061.JavaMail.Joan@RITA>
In-Reply-To: <27966981.1153.1473829264061.JavaMail.Joan@RITA>
Accept-Language: en-US
Content-Language: en-US
received-spf: SoftFail (protection.outlook.com: domain of transitioning
 hotmail.com discourages use of 10.152.80.54 as permitted sender)
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [HMe+jp+bRK6Mh3JjZkqyd77JY3kGlzXL]
x-incomingheadercount: 38
x-eopattributedmessage: 0
x-microsoft-exchange-diagnostics: 1;CO1NAM03HT024;6:aFxF0INZsJjhst4fDE5rzUzgGxkVZhtqW9uEw3TiTO+nQ+LtvJRS3+I8rrC4+QqA3ZgEzDt+SiKl5JyfFfcVe0dMjAbIZUnyEFTTBfUe/PnCr5f78jZCoILzQT6H5Y/124jdysULQ1Vs78YOtBIB59uE/0plOgGVcpJLuJHvbvP0FTdqTUp6sYJmCbZeT+AE2MQbNAsSiEebXibAOOQKOLrIhxhxNoBKkdSsRiob8NPy9MCWddNGlNkHPNk0JKEhiSPnOO1BiEywRJh2dP3adqACJChGb4BJgz0vCDoO8TI=;5:rfOc357q+Q6Sm4CDcdS0gg3k9MORaNk+d30acwuuFIvwvfQPZlUPTZZrbuWRLl7Vkanwc1VYEBHw/ZvnegF+00vuOTQqCqBNMMmA9WjbBzi8lNxEixgGsj2X4/g8HDaiRkrb/qYK9p58QEw35B+vyQ==;24:JJop/SSaHvnDaHGPSQJtUTKlI390FhH2JkVw5PBkkfsOJ3q+YbdrfJmJ4Dm97hbyxB+WcnH4my99B6tDw0XddK7IUQe1jbv0w6k/mQNQUik=;7:saoolbEoE7k+eCYC7619YToDQrYGtdNpugTY/wZER1i34x1Q8BIrW3FB/c2Uvbgmn6e06lPjrztSqcFQ3WpPjUraHsY5D5CWDQio6u7cMru23Ayt0/juoY5RVx24C1w9WDkKD2r42CUBR4Caep/8x1vvZXRNgOYlOQ4kuZaGX52qAsMGkNwGuJsZ1PLZNz2+fhPcgjozcGniBBS4V5GN0/bhm8V07e7MMMeHO3bCskd/52TuFqk/vEFWdFz5VwE7
x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(10019020)(98900003);DIR:OUT;SFP:1102;SCL:1;SRVR:CO1NAM03HT024;H:BN4PR10MB0995.namprd10.prod.outlook.com;FPR:;SPF:None;LANG:en;
x-ms-office365-filtering-correlation-id: 56d2c61d-7bb2-418f-29ab-08d3dc8e290e
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(1601124038)(1603103081)(1601125047);SRVR:CO1NAM03HT024;
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(432015012)(82015046);SRVR:CO1NAM03HT024;BCL:0;PCL:0;RULEID:;SRVR:CO1NAM03HT024;
x-forefront-prvs: 006546F32A
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_BN4PR10MB0995D26B5AF047D8386F2A0CA2F10BN4PR10MB0995namp_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Sep 2016 10:59:12.1002
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1NAM03HT024
X-OriginalArrivalTime: 14 Sep 2016 10:59:13.0985 (UTC) FILETIME=[07AD2310:01D20E77]
archived-at: Wed, 14 Sep 2016 10:59:28 -0000

--_000_BN4PR10MB0995D26B5AF047D8386F2A0CA2F10BN4PR10MB0995namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Same thing is happening on a Linux cluster. :( It even causes the beam proc=
ess to crash.

I am using free tier instances on AWS, with 1 Core and 1GB RAM. Would that =
cause this issue?


________________________________
From: Joan Touzet <wohali@apache.org>
Sent: Wednesday, September 14, 2016 4:59 AM
To: dev@couchdb.apache.org
Subject: Re: High CPU usage caused by clustering and/or replication?

I would recommend that you try and reproduce the scenario you're
running into with a Linux-based cluster, and if you find it there
as well, file a bug with us so we can try and work your problem out.

If the problem goes away, then yes, I recommend for now you consider a
Linux cluster. ;)

I also recommend you use a minimum of 3 nodes for a cluster of q=3D8 (the
default).

----- Original Message -----
> From: "Joey Samonte" <csharpdeveloper@hotmail.com>
> To: dev@couchdb.apache.org, "Joan Touzet" <wohali@apache.org>
> Sent: Wednesday, September 14, 2016 12:04:04 AM
> Subject: Re: High CPU usage caused by clustering and/or replication?
>
> I actually stopped all replications and the high CPU occurs on the
> 2nd node. I have only 2 nodes in the cluster.
>
> We really want to move to CouchDB 2.0, but we can't afford this issue
> I am encountering.
>
> Do you suggest we just use Linux-based cluster? :(
>
>
>
> ________________________________
> From: Joan Touzet <wohali@apache.org>
> Sent: Wednesday, September 14, 2016 3:47 AM
> To: dev@couchdb.apache.org
> Subject: Re: High CPU usage caused by clustering and/or replication?
>
> Joey,
>
> I'm sorry, I can't reproduce this problem. If you're able to provide
> sample data that causes this issue, I'd be pleased to test (on a
> non-virtualized Windows machine) with that.
>
> I will note that I am only testing with a local single CouchDB
> server,
> not in a cluster.  What if you try replicating to a non-clustered
> Windows server? We have absolutely zero testing done on CouchDB
> clustered on Windows, and in fact really should declare our support
> for
> that configuration as beta. I will state I have not set up a local
> Windows-based cluster (on separate Windows machines; I have run a 3
> node-on-1-server development cluster setup) so I can't guarantee that
> this isn't the problem.
>
> Please note this Known Issue in CouchDB 2.0:
>   https://issues.apache.org/jira/browse/COUCHDB-2980
>
> To quote our Known Issues document: "for now, we recommend always
> using
> the URL format for both source and target specifications."
>
> Finally, you should know it was my decision to continue Windows
> support
> for CouchDB, as no one else was able to step up. I did so primarily
> because, in my experience, we have developers who develop against a
> local CouchDB instance on their Windows laptops, then move to a
> Linux-hosted cluster for server use. I didn't expect people would be
> trying to leverage the Windows port in a real server configuration.
>
> That's not to say we couldn't eventually support Windows in a server
> cluster configuration as "out of beta," but it'll take more
> experience
> and more repeatable bugs at the very least - and active CouchDB
> Windows
> developers with performance tuning experience at best.
>
> -Joan
>
> ----- Original Message -----
> > From: "Joey Samonte" <csharpdeveloper@hotmail.com>
> > To: dev@couchdb.apache.org, "Joan Touzet" <wohali@apache.org>
> > Sent: Tuesday, September 13, 2016 10:53:18 PM
> > Subject: Re: High CPU usage caused by clustering and/or
> > replication?
> >
> > I have tried to stop all replications, then restarted CouchDB, but
> > still having almost 100% CPU with erl.exe
> >
> >
> > ________________________________
> > From: Joan Touzet <wohali@apache.org>
> > Sent: Friday, September 9, 2016 8:11 PM
> > To: dev@couchdb.apache.org
> > Subject: Re: High CPU usage caused by clustering and/or
> > replication?
> >
> > I don't know, a simple view with 100 documents should build quickly
> > unless your view code is unusual.
> >
> > I would recommend replicating the database to a UNIX-based 2.0
> > cluster
> > to see if the problem is Windows-specific.
> >
> > Sorry that I don't have any other ideas at the moment.
> >
> > -Joan
> >
> > ----- Original Message -----
> > > From: "Joey Samonte" <csharpdeveloper@hotmail.com>
> > > To: dev@couchdb.apache.org
> > > Sent: Friday, September 9, 2016 3:08:37 PM
> > > Subject: RE: High CPU usage caused by clustering and/or
> > > replication?
> > >
> > > Hi ma'am Joan,
> > >
> > > But if the cause is a view being rebuilt, why is it so high? I
> > > only
> > > have less than 100 documents on that database which I wanted to
> > > test
> > > initially.
> > >
> > > > Date: Fri, 9 Sep 2016 07:10:35 -0400
> > > > From: wohali@apache.org
> > > > To: dev@couchdb.apache.org
> > > > Subject: Re: High CPU usage caused by clustering and/or
> > > > replication?
> > > >
> > > > Hi Joey,
> > > >
> > > > I'm going to oversimplify a bit here to make the explanation
> > > > easier.
> > > >
> > > > If you check the process manager/top on each node, do you see
> > > > couchjs
> > > > responsible for that 100% CPU?
> > > >
> > > > Remember that views are not replicated, only documents. So if
> > > > you
> > > > have
> > > > design documents that you replicated, those views may be being
> > > > rebuilt.
> > > >
> > > > So-called "internal replication" will be placing segments of
> > > > the
> > > > database
> > > > onto each node, and view building will be happening on each
> > > > node
> > > > as
> > > > well.
> > > > Since the view is also sharded, each shard of the database gets
> > > > its
> > > > own
> > > > view shard; that has to be calculated independently on each
> > > > node.
> > > >
> > > > Even if you're not building views, remember that a cluster
> > > > places
> > > > part
> > > > of the database on each node, spreading the load out as equally
> > > > as
> > > > possible. In a standard 3-node cluster, each document write
> > > > will
> > > > be
> > > > written redundantly to every node, meaning CPU load will be
> > > > (mostly)
> > > > equal across all 3 nodes as you write that document.
> > > >
> > > > Hope this helps,
> > > > Joan
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Joey Samonte" <csharpdeveloper@hotmail.com>
> > > > > To: dev@couchdb.apache.org
> > > > > Sent: Thursday, September 8, 2016 11:30:46 PM
> > > > > Subject: High CPU usage caused by clustering and/or
> > > > > replication?
> > > > >
> > > > > Good day,
> > > > >
> > > > > I have setup a cluster with two nodes, and I am replicating
> > > > > the
> > > > > first
> > > > > node to an existing database through a loadbalancer. But I am
> > > > > getting a 100% CPU usage after a few minutes on both nodes.
> > > > > Is
> > > > > this
> > > > > an issue with RC4?
> > > > >
> > > > > Thanks.
> > > > >
> > >
> >
>

--_000_BN4PR10MB0995D26B5AF047D8386F2A0CA2F10BN4PR10MB0995namp_--