tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lisa Woodring <lisa.woodr...@iglass.net>
Subject High thread count & load on Tomcat8 when accessing AJP port with no request
Date Tue, 18 Nov 2014 16:52:52 GMT
We recently upgraded from Tomcat 6.0.29 to Tomcat 8.0.14.  Everything
appears to be working fine, except that Tomcat is keeping a high # of
threads (in TIMED_WAITING state) -- and the CPU has a high load & low
idle time.  We are currently running Tomcat8 on 2 internal test
machines, where we also monitor their statistics.  In order to monitor
the availability of the HTTPS/AJP port (Apache-->Tomcat), our
monitoring software opens a port to verify that this works -- but then
does not follow that up with an actual request.  This happens every 2
minutes.  We have noticed that the high thread/load activity on Tomcat
coincides with this monitoring.  If we disable our monitoring, the
issue does not happen.  We have enabled/disabled the monitoring on
both machines over several days (and there is only very minimal,
sometimes non-existent) internal traffic otherwise) -- in order to
verify that the monitoring is really the issue.  Once these threads
ramp up, they stay there or keep increasing.  We had no issues running
on Tomcat 6 (the thread count stayed low, low load, high idle time).

The thread backtraces for these threads look like this:
-----------------------------------------------------------------------------
Thread[catalina-exec-24,5,main]
         at sun.misc.Unsafe.park(Native Method)
         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
         at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
         at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
         at java.lang.Thread.run(Thread.java:745)
-----------------------------------------------------------------------------
The thread count grows over time (goes up to 130-150 threads after 2
hours).  Setting 'connectionTimeout' (as opposed to the default of
never timing out) does seems to help "some" -- the # of threads isn't
quite as bad (only 60-80 threads after 2 hours).  However, the CPU
Idle % is still not good -- was only 10% idle with default tomcat
settings, is something like 40% idle with current settings.  Also
tried setting Apache's 'KeepAliveTimeout = 5' (currently set to 15)
but this did not make any difference.


Is there some configuration we can set to make Tomcat tolerant of this
monitoring?  (We have tried setting connectionTimeout &
keepAliveTimeout on the Connector.  And we have tried putting the
Connector behind an Executor with maxIdleTime.)
OR, should we modify our monitoring somehow?  And if so, suggestions?


* Running on Linux CentOS release 5.9
* running Apache in front of Tomcat for authentication, using mod_jk
* Tomcat 8.0.14

relevant sections of tomcat/conf/server.xml:
------------------------------------------------------------------------
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
               maxThreads="250" minSpareThreads="20" maxIdleTime="60000" />

    <Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000" redirectPort="8443" />

    <Connector executor="tomcatThreadPool" port="8009" protocol="AJP/1.3"
               redirectPort="8443" maxThreads="256"
               connectionTimeout="3000" keepAliveTimeout="60000" />
----------------------------------------------------------------------------

If interested, I can provide graphing of the machine's thread count,
cpu idle%, and cpu load.
Any suggestions would be most welcome.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message