tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: Tomcat 8.5 NIO w/ SSL Windows CPU issues
Date Wed, 10 Aug 2016 15:46:49 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mike,

On 8/10/16 6:15 AM, Mike Noordermeer wrote:
> Hi,
> 
> After an upgrade to Tomcat 8.5, we are experiencing an issue where 
> Tomcat starts generating a high CPU load (100%), probably after an 
> HTTP network scan. The bug seems to be related to Windows, NIO and 
> possibly SSL. I have a Yourkit dump and several thread dumps that
> show the issue, and was wondering if anyone is interested in this,
> and if we can gather any extra information to help debug this
> issue.
> 
> Setup: Windows 2k8r2, Tomcat 8.5.4, Java 8u102, NIO HTTP and NIO
> JSSE HTTPS connector.
> 
> Out of nothing, Tomcat starts using 100% CPU. I made some thread 
> dumps, available here:
> 
> https://gist.github.com/MikeN123/f4a85f09231cfda7a9e632b64f27dcdc 
> https://gist.github.com/MikeN123/7dfe17ae95b8d516d86e0d7126cbaa02 
> https://gist.github.com/MikeN123/750da8580e04e0498f70b81dbd1a5c52 
> https://gist.github.com/MikeN123/2e83307b7c1216339d4fa73b30c02f1a 
> https://gist.github.com/MikeN123/8850ef2a60d39a4dc140b2d8fef18c3f
> 
> I also have some Yourkit stats available, but as these may contain 
> confidential information, I won't share them in public. Basically, 
> what we see is that the thread https-jsse-nio-443-ClientPoller-0
> is continuously runnable and using CPU on 
> sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(), and various
> other https-jsse-nio-443-exec threads are waiting (parked) or
> running. These threads together take up all the CPU. A Yourkit
> thread view showing the issue starting around 11:02:
> https://dl.eveoh.nl/yc_fal.png
> 
> We _suspect_ the issue is triggered by an HTTP scan, which
> generates the following requests in the access log, but we are
> still trying to confirm this:
> https://gist.github.com/MikeN123/581d1f17aae100f06b8c65b86870a64a
> 
> Also, we are trying to confirm whether or not NIO2 shows the same
> behaviour.
> 
> The behaviour seems to be the same as in this tomcat-users thread: 
> https://mail-archives.apache.org/mod_mbox/tomcat-users/201604.mbox/%3C
CAE-ydNF84pnoX2tP8BJ4vQisabgycP0y2vpnmjNhddz9+BKp=w@mail.gmail.com%3E
>
>  A similar issue is mentioned for some other products, but I'm not
> sure if there's a relation:
> 
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=357240 
> https://developer.jboss.org/thread/240618?start=0&tstart=0 
> https://github.com/netty/netty/issues/3857
> 
> Our next steps are:
> 
> - Switching the production site to NIO2, to see if that fixes the
> issue - Checking if we can reproduce the issue by triggering the
> HTTP vulnerability scan manually
> 
> Any ideas or requests for more information are more than welcome.

Are you fronting with a web server/reverse proxy? Those "-" requests
looks suspiciously like the kinds of requests that Apache httpd makes
to itself to verify that worker threads are still available for
certain things.

Maybe that's a way that HTTP scanners are trying to avoid detection:
by looking like "normal" stuff in the logs.

I'm curious... why are the requests coming from "10.xxx"... isn't that
within your own network? Shouldn't you KNOW what that stuff is?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJXq0xpAAoJEBzwKT+lPKRYu/sP/RdIRuBnSq36wHr8eYj6IAc9
xJmlcjdfBB4CL7DYwDvH+fddDb/W8k48ksQIk/nhvrqf0mP7V7g21L9pS3Cl/ki4
i/pKR6WhxPEkcKGZh7uxlD6bPUpJCVcvDX0W8jfl+yXXX5eJKVWwrfJHxZ7i4hEL
c8Vz/JUyXfAATWbfkiBGRhrcSqA3QG9WD1D6XZxAl6saJgqUD2tn7CbXJ+XG6GZB
563qD/fBg20FG5xq5atXhnjAl4icmhIEcaszjsgNb7cwzwPKV2PBcgF+V6ObGvHA
+5v0by0jGpaX6I5wozjVMIOpjdcQLFApimdDS4yBaqP400jKrFsbwVVjYn1E08oK
n9lxW2yfzC1l9AmbnHa2OcpRSyFVV9XyZdgkElVETz8XQbJu+E18gK9eTKufweYA
PDUVmNwXg/WaCNpZtuG8kvgsP93v4mfTzsru9I5ktMSdSti5cJPtrkHGnZicBZNB
5oWQzgtc38exUyJg9WKalAZMNiAR1WlH4mxCCDmS8TAUd0JzKexRRsQYRDKYQCJY
SU3iW5zvj5n1dcloMdyx1BbP6pGIK/FvP9Jxm4oKhDfN4WnrFHa/eYWam1B7lpO/
sgu7KJe815zgGOZS8H5cstegY2SAsi48zG1JLQut+WZE/2fBkozZVqdj/D/ybd4n
WPi5Lf4v5RcbL5w1f6OB
=19NA
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message