tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lisa Woodring <lisa.woodr...@iglass.net>
Subject Re: High thread count & load on Tomcat8 when accessing AJP port with no request
Date Wed, 19 Nov 2014 18:20:59 GMT
On Tue, Nov 18, 2014 at 2:26 PM, André Warnier <aw@ice-sa.com> wrote:
> Lisa Woodring wrote:
> ...
>> In order to monitor
>> the availability of the HTTPS/AJP port (Apache-->Tomcat), our
>> monitoring software opens a port to verify that this works -- but then
>> does not follow that up with an actual request.  This happens every 2
>> minutes.
> ...
>
> This sounds like the perfect recipe for simulating a DOS attack.  Your
> monitoring system is forcing Tomcat to allocate a thread to process the
> request which should subsequently arrive on that connection, yet that
> request never comes; so basically this thread is wasted, until the
> ConnectionTimeout triggers (after 20 seconds, according to your HTTP
> connector settings).
>
> ...
>>
>> The thread count grows over time (goes up to 130-150 threads after 2
>> hours).  Setting 'connectionTimeout' (as opposed to the default of
>> never timing out) does seems to help "some"
>
>
> Have you tried setting it shorter ? 20000 = 20000 ms = 20 seconds. That is
> still quite long if you think about a legitimate browser/application making
> a connection, and then sending a request on that connection.  Why would it
> wait so long ? A browser would never do that : it would open a connection to
> the server when it needs to send a request, and then send the request
> immediately, as soon as the connection is established.
>
> In other words : anything which opens a HTTP connection to your server, and
> then waits more than 1 or 2 seconds before sending a request on that
> connection, is certainly not a browser.
> And it probably is either a program designed to test or attack your server,
> or else a badly-designed monitoring system.. ;-)
>


The monitoring software is going thru Apache to AJP connector in
Tomcat.  As I described, with the default of no timeout, the # of
threads were much higher.  I currently have the AJP connectionTimeout
set to 3 seconds.



>
> -- the # of threads isn't
>>
>> quite as bad (only 60-80 threads after 2 hours).  However, the CPU
>> Idle % is still not good -- was only 10% idle with default tomcat
>> settings, is something like 40% idle with current settings.  Also
>> tried setting Apache's 'KeepAliveTimeout = 5' (currently set to 15)
>> but this did not make any difference.
>
>
> Note : this value is in milliseconds. setting it to 5 or 15 is almost
> equivalent to disabling keep-alive altogether. 3000 may be a reasonable
> value.


No, Apache's configuration is in seconds.


>
> KeepAlive only happens after at least one request has been received and
> processed, waiting for another (possible) request on the same connection.
> If there is never any request sent on that connection, then it would not
> apply here, and only the connectionTimeout would apply.
>
> Note that my comments above are relative to your HTTP Connector.
> For the AJP Connector, other circumstances apply.
>
> If you are using AJP, it implies that there is a front-end server, using a
> module such as mod_jk or mod_proxy_ajp to connect to Tomcat's AJP Connector.
> In that case, you should probably leave Tomcat's connectionTimeout to its
> default value, and let the front-end server handle such things as the
> connection timeout and the keep-alive timeout.  The connector module on the
> front-end server will manage these connections to Tomcat, and it may
> pre-allocate some connections, to constitute a pool of available connections
> for when it actually does need to send a request to Tomcat over one such
> connection.  Timing out these connections at the Tomcat level may thus be
> contra-productive, forcing the front-end to re-create them constantly.
>
>>


Yes, as I stated, Apache is running in front of Tomcat using mod_jk.
My big question is why is this now an issue?  This monitoring software
has been running for years now.  It has only been an issue since we
upgraded to Tomcat 8.

I also forgot to mention that we are using APR.



>>
>> Is there some configuration we can set to make Tomcat tolerant of this
>> monitoring?  (We have tried setting connectionTimeout &
>> keepAliveTimeout on the Connector.  And we have tried putting the
>> Connector behind an Executor with maxIdleTime.)
>> OR, should we modify our monitoring somehow?  And if so, suggestions?
>>
>
> I would think so.  Have your monitoring send an actual request to Tomcat
> (and read the response); even a request that results in an error would
> probably be better than no request at all.  But better would be to request
> something real but small, which at the Tomcat level would be efficient to
> respond to (e.g. not a 5 MB image file).
> Create a little webapp which just responds "I'm fine" (*), and check that
> response in your monitor.  It will tell you not only that Tomcat has opened
> the port, but also that Tomcat webapps are actually working (and how quickly
> it answers).
> And do not try to monitor the AJP port directly. Monitor a request to the
> front-end, which should arrive to Tomcat via the AJP port.  The AJP
> connector module on the front-end will respond with its own error, if it
> cannot talk to Tomcat.
>
> (*) actually, there may even exist some built-in mechanism in Tomcat,
> designed precisely for such kind of usage (or at least usable for it).
> Any of the experts on the list ? does the standard vanilla Tomcat offer some
> URL which can be called, and triggers some small efficient response readable
> by a monitoring program ?
>
>
>
>
>
> ...
>>
>>
>> * Running on Linux CentOS release 5.9
>> * running Apache in front of Tomcat for authentication, using mod_jk
>> * Tomcat 8.0.14
>>
>> relevant sections of tomcat/conf/server.xml:
>> ------------------------------------------------------------------------
>>     <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>                maxThreads="250" minSpareThreads="20" maxIdleTime="60000"
>> />
>>
>>     <Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
>>                connectionTimeout="20000" redirectPort="8443" />
>>
>>     <Connector executor="tomcatThreadPool" port="8009" protocol="AJP/1.3"
>>                redirectPort="8443" maxThreads="256"
>>                connectionTimeout="3000" keepAliveTimeout="60000" />
>>
>> ----------------------------------------------------------------------------
>>
>> If interested, I can provide graphing of the machine's thread count,
>> cpu idle%, and cpu load.
>> Any suggestions would be most welcome.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message