tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lisa Woodring <lisa.woodr...@iglass.net>
Subject Re: High thread count & load on Tomcat8 when accessing AJP port with no request
Date Wed, 19 Nov 2014 18:28:57 GMT
On Wed, Nov 19, 2014 at 1:20 PM, Lisa Woodring <lisa.woodring@iglass.net> wrote:
> On Tue, Nov 18, 2014 at 2:26 PM, André Warnier <aw@ice-sa.com> wrote:
>> Lisa Woodring wrote:
>> ...
>>> In order to monitor
>>> the availability of the HTTPS/AJP port (Apache-->Tomcat), our
>>> monitoring software opens a port to verify that this works -- but then
>>> does not follow that up with an actual request.  This happens every 2
>>> minutes.
>> ...
>>
>> This sounds like the perfect recipe for simulating a DOS attack.  Your
>> monitoring system is forcing Tomcat to allocate a thread to process the
>> request which should subsequently arrive on that connection, yet that
>> request never comes; so basically this thread is wasted, until the
>> ConnectionTimeout triggers (after 20 seconds, according to your HTTP
>> connector settings).
>>
>> ...
>>>
>>> The thread count grows over time (goes up to 130-150 threads after 2
>>> hours).  Setting 'connectionTimeout' (as opposed to the default of
>>> never timing out) does seems to help "some"
>>
>>
>> Have you tried setting it shorter ? 20000 = 20000 ms = 20 seconds. That is
>> still quite long if you think about a legitimate browser/application making
>> a connection, and then sending a request on that connection.  Why would it
>> wait so long ? A browser would never do that : it would open a connection to
>> the server when it needs to send a request, and then send the request
>> immediately, as soon as the connection is established.
>>
>> In other words : anything which opens a HTTP connection to your server, and
>> then waits more than 1 or 2 seconds before sending a request on that
>> connection, is certainly not a browser.
>> And it probably is either a program designed to test or attack your server,
>> or else a badly-designed monitoring system.. ;-)
>>
>
>
> The monitoring software is going thru Apache to AJP connector in
> Tomcat.  As I described, with the default of no timeout, the # of
> threads were much higher.  I currently have the AJP connectionTimeout
> set to 3 seconds.


Actually, I received a little clarification on the monitoring software
(I didn't write it).  What it's trying to test is that the AJP port
itself is actually accepting connections.  With Apache in front in a
production system, it could forward the actual request to one of
several Tomcat boxes -- but we don't know which one from the outside.
The monitoring software is trying to test -- for each Tomcat instance
-- if it is accepting connections.  It used to send an "nmap" request,
but now sends essentially a "tcp ping" -- gets a response & moves on.

My main questions are:
1) Why was this ok on Tomcat 6?  but now an issue with Tomcat 8?
2) Suggestions on how to monitor this better?


>
>
>
>>
>> -- the # of threads isn't
>>>
>>> quite as bad (only 60-80 threads after 2 hours).  However, the CPU
>>> Idle % is still not good -- was only 10% idle with default tomcat
>>> settings, is something like 40% idle with current settings.  Also
>>> tried setting Apache's 'KeepAliveTimeout = 5' (currently set to 15)
>>> but this did not make any difference.
>>
>>
>> Note : this value is in milliseconds. setting it to 5 or 15 is almost
>> equivalent to disabling keep-alive altogether. 3000 may be a reasonable
>> value.
>
>
> No, Apache's configuration is in seconds.
>
>
>>
>> KeepAlive only happens after at least one request has been received and
>> processed, waiting for another (possible) request on the same connection.
>> If there is never any request sent on that connection, then it would not
>> apply here, and only the connectionTimeout would apply.
>>
>> Note that my comments above are relative to your HTTP Connector.
>> For the AJP Connector, other circumstances apply.
>>
>> If you are using AJP, it implies that there is a front-end server, using a
>> module such as mod_jk or mod_proxy_ajp to connect to Tomcat's AJP Connector.
>> In that case, you should probably leave Tomcat's connectionTimeout to its
>> default value, and let the front-end server handle such things as the
>> connection timeout and the keep-alive timeout.  The connector module on the
>> front-end server will manage these connections to Tomcat, and it may
>> pre-allocate some connections, to constitute a pool of available connections
>> for when it actually does need to send a request to Tomcat over one such
>> connection.  Timing out these connections at the Tomcat level may thus be
>> contra-productive, forcing the front-end to re-create them constantly.
>>
>>>
>
>
> Yes, as I stated, Apache is running in front of Tomcat using mod_jk.
> My big question is why is this now an issue?  This monitoring software
> has been running for years now.  It has only been an issue since we
> upgraded to Tomcat 8.
>
> I also forgot to mention that we are using APR.
>
>
>
>>>
>>> Is there some configuration we can set to make Tomcat tolerant of this
>>> monitoring?  (We have tried setting connectionTimeout &
>>> keepAliveTimeout on the Connector.  And we have tried putting the
>>> Connector behind an Executor with maxIdleTime.)
>>> OR, should we modify our monitoring somehow?  And if so, suggestions?
>>>
>>
>> I would think so.  Have your monitoring send an actual request to Tomcat
>> (and read the response); even a request that results in an error would
>> probably be better than no request at all.  But better would be to request
>> something real but small, which at the Tomcat level would be efficient to
>> respond to (e.g. not a 5 MB image file).
>> Create a little webapp which just responds "I'm fine" (*), and check that
>> response in your monitor.  It will tell you not only that Tomcat has opened
>> the port, but also that Tomcat webapps are actually working (and how quickly
>> it answers).
>> And do not try to monitor the AJP port directly. Monitor a request to the
>> front-end, which should arrive to Tomcat via the AJP port.  The AJP
>> connector module on the front-end will respond with its own error, if it
>> cannot talk to Tomcat.
>>
>> (*) actually, there may even exist some built-in mechanism in Tomcat,
>> designed precisely for such kind of usage (or at least usable for it).
>> Any of the experts on the list ? does the standard vanilla Tomcat offer some
>> URL which can be called, and triggers some small efficient response readable
>> by a monitoring program ?
>>
>>
>>
>>
>>
>> ...
>>>
>>>
>>> * Running on Linux CentOS release 5.9
>>> * running Apache in front of Tomcat for authentication, using mod_jk
>>> * Tomcat 8.0.14
>>>
>>> relevant sections of tomcat/conf/server.xml:
>>> ------------------------------------------------------------------------
>>>     <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>>                maxThreads="250" minSpareThreads="20" maxIdleTime="60000"
>>> />
>>>
>>>     <Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
>>>                connectionTimeout="20000" redirectPort="8443" />
>>>
>>>     <Connector executor="tomcatThreadPool" port="8009" protocol="AJP/1.3"
>>>                redirectPort="8443" maxThreads="256"
>>>                connectionTimeout="3000" keepAliveTimeout="60000" />
>>>
>>> ----------------------------------------------------------------------------
>>>
>>> If interested, I can provide graphing of the machine's thread count,
>>> cpu idle%, and cpu load.
>>> Any suggestions would be most welcome.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message