tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: Monitoring Connections
Date Tue, 13 Oct 2015 13:34:33 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Jamie,

On 10/9/15 10:03 AM, Jamie Jackson wrote:
> Here's the stack trace dump: 
> https://gist.github.com/jamiejackson/ca2a49d2c8afac496067
> 
> FWIW, I've been trying to come up with a reliable test case to
> trigger the problem, but I haven't nailed it yet. I've suspected
> that it's related to file (large or slow) HTTP file uploads, and
> that's what I was running at the time, which helps to explain the
> java.net.SocketInputStream.socketRead0 traces.

Your server looks mostly idle. It also looks like it's running both
Tomcat and Jetty in the same instance. What's going on?

> "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native) 
> java.lang.Thread.State: RUNNABLE at
> java.net.SocketInputStream.socketRead0(Native Method) at
> java.net.SocketInputStream.read(SocketInputStream.java:152) at
> java.net.SocketInputStream.read(SocketInputStream.java:122) at
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535)
>
> 
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
> at
> org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566)
>
> 
at
org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137)
> at
> org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348)
>
> 
at org.apache.coyote.Request.doRead(Request.java:422)
> at
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
>
> 
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390)
> at
> org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304)
>
> 
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106)
> at
> com.intergral.fusionreactor.j2ee.filter.surrogate.FusionReactorServletRequestProxy$1.read(FusionReactorServletRequestProxy.java:123)
>
> 
at java.io.InputStream.read(InputStream.java:179)
> at
> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977)
>
> 
at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) 
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334) -
> locked java.io.BufferedInputStream@6e81abfc at
> java.io.FilterInputStream.read(FilterInputStream.java:107)

Obviously, this thread is reading a file upload. Also these:

"http-bio-8888-exec-21" Id=27309 RUNNABLE (in native)
   java.lang.Thread.State: RUNNABLE
"http-bio-8888-exec-26" Id=27314 RUNNABLE (in native)
   java.lang.Thread.State: RUNNABLE

This thread is making an outgoing HTTP request, which could be hanging:

> "Thread-27550" Id=27623 RUNNABLE (in native) 
> java.lang.Thread.State: RUNNABLE at
> java.net.SocketInputStream.socketRead0(Native Method) at
> java.net.SocketInputStream.read(SocketInputStream.java:152) at
> java.net.SocketInputStream.read(SocketInputStream.java:122) at
> sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at
> sun.security.ssl.InputRecord.read(InputRecord.java:480) at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) -
> locked java.lang.Object@659eed19 at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
>
> 
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
> - locked sun.security.ssl.AppInputStream@7c98d0ab at
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>
> 
at
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> at
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>
> 
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>
> 
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>
> 
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> at
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>
> 
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>
> 
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>
> 
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>
> 
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> 
at java.lang.reflect.Method.invoke(Method.java:606)
> at
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.execute(HTTPEngine4Impl.java:416)
>
> 
at
lucee.commons.net.http.httpclient4.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:252)
> at
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.get(HTTPEngine4Impl.java:112)
>
> 
at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:86)
> at
> lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:108)
>
> 
at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:58)


Same with this one:

"Thread-27551" Id=27624 RUNNABLE (in native)
   java.lang.Thread.State: RUNNABLE

But neither of the above threads making HTTP requests are
request-processing threads: they are independent and shouldn't block
incoming requests.

You have 12 total request-processing threads: the ones named
"http-bio-8888-exec-[#]". Most of them aren't doing anything -- just
waiting on a new request to come in.

You have an AJP connector defined, but it looks like it's not being used.

Initially, you said that you used mod_proxy from httpd. Are you
expecting to use mod_proxy_http or mod_proxy_ajp? It certainly looks
like you are using the HTTP flavor of mod_proxy since your
file-uploads are going to the HTTP connector on port 8888 instead of
the AJP connector on port 8009. If you don't need the AJP connector,
you might want to disable it and get a small amount of resources back.

When you try to make a connection during these incidents, do you get
any errors on the httpd side? I know mod_jk will complain if it can't
make a connection or if there is a timeout... I suspect mod_proxy_http
will do the same.

Can you post your <Connector> configuration from conf/server.xml?
Remember to remove any sensitive information that you might have in there.

Also, what do your ProxyPass/ProxyPassReverse configuration directives
look like in httpd.conf?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJWHQhoAAoJEBzwKT+lPKRYoVwP/08YV4m1q3jXp+ixsPaPReEf
m9wnqPlWpX9tGjwVcsYvnqMPTJe8Ew+yzIFZ5NkPsmQk/eeKQFMYBsr0yDje4Nl0
Suk+OfNXhdL7RUn/PZg1Rd1xl7hcVeJmgOD6HRIcXAsnB+iTpCYpR+FtTnoGosia
PQ64AKEZR7KliP0bkM04NY4hHa+eA5vp2kGyGAAkA2g+sxUUXgiAVoUYMbLYwetv
CKqwlCFcG+JF9M2qi4n8vVwpsTqdLrInlO+yAiejZoQuHaSefnolIkFE0y+08bnz
Yv1nhNXwIOJAIvsl2BFoQIXXnpVh2ncILLIzhrcBBsazBy0RxccKsOJR0F6SKEWg
3JLD6z/3CxtuW28KU2nFIQiKEQHzhZ+fV1NAp3XrHaQPncFGlNGPOUgy4Ggqxy3j
eAOkN1si3GFRm+ILELDO4ekKQCoAAdWs3MMpVTOKdL6mMO1OiYGPHzVTclgLloxc
o6FVgy81s6Hk9UJlu5enZJu9I+5tFxlD1gXrBmG7ryCOyiDn9pHmZ68EBpe3j+Zf
HPUtt7XP5eTVZp8vlVS3ebBHjrMkxUYGuBKYZF7prGfA5LE42nctdMslneiyZJJc
AOrcp32bgJFCcXmdBovD2UrJu0Evc4su3vV4JW9PuFSE1i3ntFLKuqL4FiArf1ls
+0KkHZDkU8C3MP3MCySA
=5Mun
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message