tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lisa Woodring <lisa.woodr...@iglass.net>
Subject Re: High thread count & load on Tomcat8 when accessing AJP port with no request
Date Wed, 19 Nov 2014 18:36:27 GMT
On Tue, Nov 18, 2014 at 2:43 PM, Christopher Schultz
<chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Lisa,
>
> On 11/18/14 11:52 AM, Lisa Woodring wrote:
>> We recently upgraded from Tomcat 6.0.29 to Tomcat 8.0.14.
>> Everything appears to be working fine, except that Tomcat is
>> keeping a high # of threads (in TIMED_WAITING state) -- and the CPU
>> has a high load & low idle time.  We are currently running Tomcat8
>> on 2 internal test machines, where we also monitor their
>> statistics.  In order to monitor the availability of the HTTPS/AJP
>> port (Apache-->Tomcat), our monitoring software opens a port to
>> verify that this works -- but then does not follow that up with an
>> actual request.  This happens every 2 minutes.  We have noticed
>> that the high thread/load activity on Tomcat coincides with this
>> monitoring.  If we disable our monitoring, the issue does not
>> happen.  We have enabled/disabled the monitoring on both machines
>> over several days (and there is only very minimal, sometimes
>> non-existent) internal traffic otherwise) -- in order to verify
>> that the monitoring is really the issue.  Once these threads ramp
>> up, they stay there or keep increasing.  We had no issues running
>> on Tomcat 6 (the thread count stayed low, low load, high idle
>> time).
>>
>> The thread backtraces for these threads look like this:
>> -----------------------------------------------------------------------------
>>
>>
> Thread[catalina-exec-24,5,main]
>> at sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>
>>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>>
>>
> at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
>> at
>> org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31) at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>
>>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>> at java.lang.Thread.run(Thread.java:745)
>> -----------------------------------------------------------------------------
>>
>>
> The thread count grows over time (goes up to 130-150 threads after 2
>> hours).  Setting 'connectionTimeout' (as opposed to the default of
>> never timing out) does seems to help "some" -- the # of threads
>> isn't quite as bad (only 60-80 threads after 2 hours).  However,
>> the CPU Idle % is still not good -- was only 10% idle with default
>> tomcat settings, is something like 40% idle with current settings.
>> Also tried setting Apache's 'KeepAliveTimeout = 5' (currently set
>> to 15) but this did not make any difference.
>>
>>
>> Is there some configuration we can set to make Tomcat tolerant of
>> this monitoring?  (We have tried setting connectionTimeout &
>> keepAliveTimeout on the Connector.  And we have tried putting the
>> Connector behind an Executor with maxIdleTime.) OR, should we
>> modify our monitoring somehow?  And if so, suggestions?
>>
>>
>> * Running on Linux CentOS release 5.9 * running Apache in front of
>> Tomcat for authentication, using mod_jk * Tomcat 8.0.14
>>
>> relevant sections of tomcat/conf/server.xml:
>> ------------------------------------------------------------------------
>>
>>
> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>> maxThreads="250" minSpareThreads="20" maxIdleTime="60000" />
>>
>> <Connector executor="tomcatThreadPool" port="8080"
>> protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"
>> />
>>
>> <Connector executor="tomcatThreadPool" port="8009"
>> protocol="AJP/1.3" redirectPort="8443" maxThreads="256"
>> connectionTimeout="3000" keepAliveTimeout="60000" />
>
> Both of these connectors should be NIO connectors, so they should not
> block while waiting for more input. That means that you should not run
> out of threads (which is good), but those connections will sit in the
> poller queue for a long time (20 seconds for HTTP, 3 seconds for AJP)
> and then sit in the acceptor queue for the same amount of time (to
> check for a "next" keepAlive request). Are you properly shutting-down
> the connection on the client end every 2 minutes?
>



The monitoring software is trying to test is that the AJP port itself
is actually accepting connections.  With Apache in front in a
production system, it could forward the actual request to one of
several Tomcat boxes -- but we don't know which one from the outside.
The monitoring software is trying to test -- for each Tomcat instance
-- if it is accepting connections.  It used to send an "nmap" request,
but now sends essentially a "tcp ping" -- to port 8009, gets a
response & moves on.  So, no, it does not shutdown the connection --
it's pretty simple/dumb.

My main questions are:
1) Why was this ok on Tomcat 6?  but now an issue with Tomcat 8?
2) Suggestions on how to monitor this better?



The graphing I have -- are images.  Is there a good way to share these?


>> ----------------------------------------------------------------------------
>>
>>  If interested, I can provide graphing of the machine's thread
>> count, cpu idle%, and cpu load. Any suggestions would be most
>> welcome.
>
> More data is always good. Remember that the list strips attachments,
> so maybe paste it somewhere (e.g. pastebin or whatever) and provide a
> link.
>
> - -chris

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message