tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: High thread count & load on Tomcat8 when accessing AJP port with no request
Date Tue, 18 Nov 2014 19:26:39 GMT
Lisa Woodring wrote:
...
 > In order to monitor
 > the availability of the HTTPS/AJP port (Apache-->Tomcat), our
 > monitoring software opens a port to verify that this works -- but then
 > does not follow that up with an actual request.  This happens every 2
 > minutes.
...

This sounds like the perfect recipe for simulating a DOS attack.  Your monitoring system 
is forcing Tomcat to allocate a thread to process the request which should subsequently 
arrive on that connection, yet that request never comes; so basically this thread is 
wasted, until the ConnectionTimeout triggers (after 20 seconds, according to your HTTP 
connector settings).

...
> The thread count grows over time (goes up to 130-150 threads after 2
> hours).  Setting 'connectionTimeout' (as opposed to the default of
> never timing out) does seems to help "some" 

Have you tried setting it shorter ? 20000 = 20000 ms = 20 seconds. That is still quite 
long if you think about a legitimate browser/application making a connection, and then 
sending a request on that connection.  Why would it wait so long ? A browser would never 
do that : it would open a connection to the server when it needs to send a request, and 
then send the request immediately, as soon as the connection is established.

In other words : anything which opens a HTTP connection to your server, and then waits 
more than 1 or 2 seconds before sending a request on that connection, is certainly not a 
browser.
And it probably is either a program designed to test or attack your server, or else a 
badly-designed monitoring system.. ;-)


-- the # of threads isn't
> quite as bad (only 60-80 threads after 2 hours).  However, the CPU
> Idle % is still not good -- was only 10% idle with default tomcat
> settings, is something like 40% idle with current settings.  Also
> tried setting Apache's 'KeepAliveTimeout = 5' (currently set to 15)
> but this did not make any difference.

Note : this value is in milliseconds. setting it to 5 or 15 is almost equivalent to 
disabling keep-alive altogether. 3000 may be a reasonable value.

KeepAlive only happens after at least one request has been received and processed, waiting

for another (possible) request on the same connection.  If there is never any request sent

on that connection, then it would not apply here, and only the connectionTimeout would apply.

Note that my comments above are relative to your HTTP Connector.
For the AJP Connector, other circumstances apply.

If you are using AJP, it implies that there is a front-end server, using a module such as

mod_jk or mod_proxy_ajp to connect to Tomcat's AJP Connector.
In that case, you should probably leave Tomcat's connectionTimeout to its default value, 
and let the front-end server handle such things as the connection timeout and the 
keep-alive timeout.  The connector module on the front-end server will manage these 
connections to Tomcat, and it may pre-allocate some connections, to constitute a pool of 
available connections for when it actually does need to send a request to Tomcat over one

such connection.  Timing out these connections at the Tomcat level may thus be 
contra-productive, forcing the front-end to re-create them constantly.

> 
> 
> Is there some configuration we can set to make Tomcat tolerant of this
> monitoring?  (We have tried setting connectionTimeout &
> keepAliveTimeout on the Connector.  And we have tried putting the
> Connector behind an Executor with maxIdleTime.)
> OR, should we modify our monitoring somehow?  And if so, suggestions?
> 

I would think so.  Have your monitoring send an actual request to Tomcat (and read the 
response); even a request that results in an error would probably be better than no 
request at all.  But better would be to request something real but small, which at the 
Tomcat level would be efficient to respond to (e.g. not a 5 MB image file).
Create a little webapp which just responds "I'm fine" (*), and check that response in your

monitor.  It will tell you not only that Tomcat has opened the port, but also that Tomcat

webapps are actually working (and how quickly it answers).
And do not try to monitor the AJP port directly. Monitor a request to the front-end, which

should arrive to Tomcat via the AJP port.  The AJP connector module on the front-end will

respond with its own error, if it cannot talk to Tomcat.

(*) actually, there may even exist some built-in mechanism in Tomcat, designed precisely 
for such kind of usage (or at least usable for it).
Any of the experts on the list ? does the standard vanilla Tomcat offer some URL which can

be called, and triggers some small efficient response readable by a monitoring program ?





...
> 
> * Running on Linux CentOS release 5.9
> * running Apache in front of Tomcat for authentication, using mod_jk
> * Tomcat 8.0.14
> 
> relevant sections of tomcat/conf/server.xml:
> ------------------------------------------------------------------------
>     <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>                maxThreads="250" minSpareThreads="20" maxIdleTime="60000" />
> 
>     <Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
>                connectionTimeout="20000" redirectPort="8443" />
> 
>     <Connector executor="tomcatThreadPool" port="8009" protocol="AJP/1.3"
>                redirectPort="8443" maxThreads="256"
>                connectionTimeout="3000" keepAliveTimeout="60000" />
> ----------------------------------------------------------------------------
> 
> If interested, I can provide graphing of the machine's thread count,
> cpu idle%, and cpu load.
> Any suggestions would be most welcome.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message