tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier (tomcat) ...@ice-sa.com>
Subject Re: Tomcat 8.5 NIO w/ SSL Windows CPU issues
Date Wed, 10 Aug 2016 16:26:28 GMT
On 10.08.2016 17:46, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Mike,
>
> On 8/10/16 6:15 AM, Mike Noordermeer wrote:
>> Hi,
>>
>> After an upgrade to Tomcat 8.5, we are experiencing an issue where
>> Tomcat starts generating a high CPU load (100%), probably after an
>> HTTP network scan. The bug seems to be related to Windows, NIO and
>> possibly SSL. I have a Yourkit dump and several thread dumps that
>> show the issue, and was wondering if anyone is interested in this,
>> and if we can gather any extra information to help debug this
>> issue.
>>
>> Setup: Windows 2k8r2, Tomcat 8.5.4, Java 8u102, NIO HTTP and NIO
>> JSSE HTTPS connector.
>>
>> Out of nothing, Tomcat starts using 100% CPU. I made some thread
>> dumps, available here:
>>
>> https://gist.github.com/MikeN123/f4a85f09231cfda7a9e632b64f27dcdc
>> https://gist.github.com/MikeN123/7dfe17ae95b8d516d86e0d7126cbaa02
>> https://gist.github.com/MikeN123/750da8580e04e0498f70b81dbd1a5c52
>> https://gist.github.com/MikeN123/2e83307b7c1216339d4fa73b30c02f1a
>> https://gist.github.com/MikeN123/8850ef2a60d39a4dc140b2d8fef18c3f
>>
>> I also have some Yourkit stats available, but as these may contain
>> confidential information, I won't share them in public. Basically,
>> what we see is that the thread https-jsse-nio-443-ClientPoller-0
>> is continuously runnable and using CPU on
>> sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(), and various
>> other https-jsse-nio-443-exec threads are waiting (parked) or
>> running. These threads together take up all the CPU. A Yourkit
>> thread view showing the issue starting around 11:02:
>> https://dl.eveoh.nl/yc_fal.png
>>
>> We _suspect_ the issue is triggered by an HTTP scan, which
>> generates the following requests in the access log, but we are
>> still trying to confirm this:
>> https://gist.github.com/MikeN123/581d1f17aae100f06b8c65b86870a64a
>>
>> Also, we are trying to confirm whether or not NIO2 shows the same
>> behaviour.
>>
>> The behaviour seems to be the same as in this tomcat-users thread:
>> https://mail-archives.apache.org/mod_mbox/tomcat-users/201604.mbox/%3C
> CAE-ydNF84pnoX2tP8BJ4vQisabgycP0y2vpnmjNhddz9+BKp=w@mail.gmail.com%3E
>>
>>   A similar issue is mentioned for some other products, but I'm not
>> sure if there's a relation:
>>
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=357240
>> https://developer.jboss.org/thread/240618?start=0&tstart=0
>> https://github.com/netty/netty/issues/3857
>>
>> Our next steps are:
>>
>> - Switching the production site to NIO2, to see if that fixes the
>> issue - Checking if we can reproduce the issue by triggering the
>> HTTP vulnerability scan manually
>>
>> Any ideas or requests for more information are more than welcome.
>
> Are you fronting with a web server/reverse proxy? Those "-" requests
> looks suspiciously like the kinds of requests that Apache httpd makes
> to itself to verify that worker threads are still available for
> certain things.
>
> Maybe that's a way that HTTP scanners are trying to avoid detection:
> by looking like "normal" stuff in the logs.
>
> I'm curious... why are the requests coming from "10.xxx"... isn't that
> within your own network? Shouldn't you KNOW what that stuff is?

In-house webserver monitoring software ? (Nagios e.g.)


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message