Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (nike.apache.org: domain of bob.deremer@thingworx.com
 designates 207.46.163.153 as permitted sender)
From: Bob DeRemer <bob.deremer@thingworx.com>
To: Tomcat Users List <users@tomcat.apache.org>
Subject: RE: attempting to achieve 100K concurrent websocket connections on
 Tomcat 7.0.48 NIO
Thread-Topic: attempting to achieve 100K concurrent websocket connections on
 Tomcat 7.0.48 NIO
Thread-Index: Ac7XTaMMjzdlhl2lQDOaBp6FPGaxhwACq7CAAAF0E5A=
Date: Sat, 2 Nov 2013 00:02:13 +0000
Message-ID: 
 <1ab51bd3ae4e4a6fa486971ef15f0224@CO1PR06MB206.namprd06.prod.outlook.com>
References: 
 <93567dd80708400aa4cbdfcdf607a945@CO1PR06MB206.namprd06.prod.outlook.com>
 <5274362D.9010507@apache.org>
In-Reply-To: <5274362D.9010507@apache.org>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


> -----Original Message-----
> From: Mark Thomas [mailto:markt@apache.org]
> Sent: Friday, November 01, 2013 7:16 PM
> To: Tomcat Users List
> Subject: Re: attempting to achieve 100K concurrent websocket connections =
on
> Tomcat 7.0.48 NIO
>=20
> On 01/11/2013 22:00, Bob DeRemer wrote:
> > QUESTION: I'm looking for some advice on what Tomcat NIO connector
> > settings to use to support 100K concurrent websocket connections.
> > Hopefully I  can reach this goal through a combination of Tomcat NIO
> > Connector settings, and Server 2008 R2 configuration [if needed].
> >
> > BACKGROUND: We're scale testing our websocket application and looking
> > to see how many concurrent websocket connections we can get on a
> > single Tomcat instance - with the goal being 100K.  I've provided the
> > test landscape details at the bottom - all VERY BIG EC2 instances over
> > 10 GB network, so memory, CPU and network do not appear to be the
> > problem when monitoring.
> >
> > PROBLEM: The problem we are running into is that we can't seem to
> > establish even 50K connections into Tomcat.  At some point, we start
> > getting connect failures, similar to the following:
>=20
> OK. Win 2k8 R2, 16GB RAM, bunch of other stuff running, client and server=
 on
> same machine, no tuning. I get 16,313 connections before it falls over. T=
hat is
> consistent with the defaults which should allow a maximum of 16384 (given=
 I
> have other stuff running).
>=20
> (I'll commit the test to trunk shortly).
>=20
> After
> netsh int ipv4 set dynamicport tcp start=3D10000 num=3D55536
>=20
> I got as far as 25121 before I hit GC issues.
>=20
> -Xmx12G -Xms12G fixed the GC problems (well, I say fixed allowed them to =
be
> ignored would be closer)
>=20
> The next run got to 55464 connections which looks to be about the limit o=
f the
> ephemeral ports.
>=20
> > I'm hoping that someone may be able to advise what changes we might
> > make to the following Tomcat NIO connector setting that will allow
> > upwards of 100K websocket connections:
> >
> > <Connector port=3D"80"
> > protocol=3D"org.apache.coyote.http11.Http11NioProtocol"
> > connectionTimeout=3D"20000" maxConnections=3D"100000"
> > maxThreads=3D"100000" redirectPort=3D"8443" />
>=20
> As Chris said the threads are probably too high.
>=20
> You can set maxThreads=3D"-1" for unlimited connections (worth doing for
> testing).
>=20
> > TEST LANDSCAPE:
> >
> > The test landscape is all Windows Server 2008 R2 boxes running in EC2
> > and the Test Client environment:
> >
> > Our test client is a multi-threaded java client that makes use of the
> > JSR356 ClientEndpoint functionality.  We're creating 40K+ websocket
> > connections from a single test client machine.  The test client is
> > Server 2008 R2 and we have configured it to allow 50K ephemeral ports,
> > so we should be able to establish 40K+ outbound websocket connections.
> >
> > Server environment:
> >
> > *         EC2 instance (cc2.8xlarge) (60 GB, 10 GB network, 16
> > vCPUs)
> >
> > *         Server 2008 R2
> >
> > *         Tomcat 7.0.48 (trunk)
> >
> > *         Java 1.7.0_45
>=20
> Look at your memory usage. My guess based on my simple test results is th=
at
> the JVM for both Tomcat and the client isn't grabbing as much memory as i=
t
> really needs. If that is the problem then -Xms20G -Xmx30G should do the t=
rick.
>=20
> Keep in mind that Tomcat is allocating at several 8k buffers for each
> connection (client and server - I'd need to check the code to be sure how=
 many
> buffers are allocated per connection) so you are going to need a fair amo=
unt of
> RAM. The TCP buffers will need quite a lot of space too but that should b=
e
> outside of the Java Object Heap.

In monitoring the memory, it's not nearing the max available we configured =
for the JVM, but perhaps there's something going on GC wise.  I don't have =
GC logging enabled and probably should.  I haven't done much with that aspe=
ct of tuning yet.

>From a Tomcat connector standpoint, should we consider increasing any of th=
e acceptThreadCount and acceptCount values - given it's such a large machin=
e?

>=20

> Mark
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org