tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse Barnum <jsb_tom...@360works.com>
Subject What is the best connector configuration for thousands of mostly idle users?
Date Fri, 07 Feb 2014 03:11:58 GMT
Problem summary:
My nio polling threads are using too much CPU time.

Application overview:
My application has from 1,300 - 4,000 users connected at any given time. Each user sends about
200 bytes, then waits 30 seconds, then sends about 200 bytes, and this just loops for each
user.
Each user connects with SSL, and we use a long keepalive to ensure that the HTTP connection
doesn't close, so that we don't have to renegotiate SSL.

Configuration:
Ubuntu 12.0.4 with Tomcat 7.0.35, 1.75 gigs of RAM.

We are using Apache with SSL and mod_proxy_ajp to forward requests to Tomcat. It has MPM module
enabled, with 500 ThreadsPerChild, so we typically have from 3-9 Apache instances running.

> <IfModule mpm_worker_module>
>     ServerLimit         12
>     ThreadLimit         1000
> 
>     StartServers         1 
>     MinSpareThreads      25
>     MaxRequestsPerChild 0
>     MaxSpareThreads     500
>     ThreadsPerChild     500
>     MaxClients          5000
> </IfModule>
> 


> ProxyPass /WSMRegister ajp://localhost:8009/WSMRegister

We are using the AJP NIO connector on port 8009 on Tomcat with 15 worker threads:

>     <!-- Define an AJP 1.3 Connector on port 8009 -->
>     <Connector port="8009" 
>         protocol="org.apache.coyote.ajp.AjpNioProtocol" 
>         redirectPort="8443"
>         minSpareThreads="1" 
>         maxThreads="15" 
>         scheme="https"
>         secure="true"
>         URIDecoding="UTF-8"
>         proxyName="secure2.360works.com"
>         proxyPort="443" />

Problem detail:
lsof is currently showing 564 open sockets between Apache and Tomcat on port 8009, with 1,352
users connected to Apache.
The two threads consuming the most CPU time in Tomcat are "NioBlockingSelector.BlockPoller-2
/ 15" and "ajp-nio-8009-ClientPoller-0 / 25". Between them, they are taking 20% of all CPU
time for the Tomcat process. I get a few times a day when our monitoring software reports
slow response times, and I'd like to solve this.

Some guesses at solutions:
I'm guessing that the high CPU usage is because they are polling all 564 open sockets constantly?
Would it make sense to reduce the number of open sockets? I didn't configure any maximum and
I don't know how to reduce this number. I'm also concerned that reducing that might negate
any benefits by increasing the number of sockets opening and closing between ajp_mod_proxy
and the NIO AJP connector.

Maybe it's already running at optimal performance and I just need to throw hardware at it,
but it seems like a solvable problem, because the actual worker threads are not doing much
at all.

--Jesse Barnum, President, 360Works
http://www.360works.com
Product updates and news on http://facebook.com/360Works
(770) 234-9293
== Don't lose your data! http://360works.com/safetynet/ for FileMaker Server ==


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message