tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie Jackson <jamieja...@gmail.com>
Subject Re: Monitoring Connections
Date Mon, 19 Oct 2015 12:59:39 GMT
About the appearance of Jetty in the stack trace dump: It's part of
FusionReactor (the JVM monitor)--it uses Jetty to serve its interface.

Thanks,
Jamie

On Fri, Oct 16, 2015 at 3:12 PM, Jamie Jackson <jamiejaxon@gmail.com> wrote:

>
>
> On Tue, Oct 13, 2015 at 9:34 AM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Jamie,
>>
>> On 10/9/15 10:03 AM, Jamie Jackson wrote:
>> > Here's the stack trace dump:
>> > https://gist.github.com/jamiejackson/ca2a49d2c8afac496067
>> >
>> > FWIW, I've been trying to come up with a reliable test case to
>> > trigger the problem, but I haven't nailed it yet. I've suspected
>> > that it's related to file (large or slow) HTTP file uploads, and
>> > that's what I was running at the time, which helps to explain the
>> > java.net.SocketInputStream.socketRead0 traces.
>>
>> Your server looks mostly idle. It also looks like it's running both
>> Tomcat and Jetty in the same instance. What's going on?
>
>
> I'm not sure yet. I'm asking around, and I'll let you know.
>
>
>> > "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native)
>> > java.lang.Thread.State: RUNNABLE at
>> > java.net.SocketInputStream.socketRead0(Native Method) at
>> > java.net.SocketInputStream.read(SocketInputStream.java:152) at
>> > java.net.SocketInputStream.read(SocketInputStream.java:122) at
>> >
>> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535)
>> >
>> >
>> at
>>
>> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
>> > at
>> >
>> org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566)
>> >
>> >
>> at
>>
>> org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137)
>> > at
>> >
>> org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348)
>> >
>> >
>> at org.apache.coyote.Request.doRead(Request.java:422)
>> > at
>> >
>> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
>> >
>> >
>> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390)
>> > at
>> > org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304)
>> >
>> >
>> at
>>
>> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106)
>> > at
>> >
>> com.intergral.fusionreactor.j2ee.filter.surrogate.FusionReactorServletRequestProxy$1.read(FusionReactorServletRequestProxy.java:123)
>> >
>> >
>> at java.io.InputStream.read(InputStream.java:179)
>> > at
>> >
>> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977)
>> >
>> >
>> at
>>
>> org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)
>> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) -
>> > locked java.io.BufferedInputStream@6e81abfc at
>> > java.io.FilterInputStream.read(FilterInputStream.java:107)
>>
>> Obviously, this thread is reading a file upload. Also these:
>>
>> "http-bio-8888-exec-21" Id=27309 RUNNABLE (in native)
>>    java.lang.Thread.State: RUNNABLE
>> "http-bio-8888-exec-26" Id=27314 RUNNABLE (in native)
>>    java.lang.Thread.State: RUNNABLE
>>
>> This thread is making an outgoing HTTP request, which could be hanging:
>>
>> > "Thread-27550" Id=27623 RUNNABLE (in native)
>> > java.lang.Thread.State: RUNNABLE at
>> > java.net.SocketInputStream.socketRead0(Native Method) at
>> > java.net.SocketInputStream.read(SocketInputStream.java:152) at
>> > java.net.SocketInputStream.read(SocketInputStream.java:122) at
>> > sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at
>> > sun.security.ssl.InputRecord.read(InputRecord.java:480) at
>> > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) -
>> > locked java.lang.Object@659eed19 at
>> > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
>> >
>> >
>> at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
>> > - locked sun.security.ssl.AppInputStream@7c98d0ab at
>> >
>> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>> >
>> >
>> at
>>
>> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
>> > at
>> >
>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>> >
>> >
>> at
>>
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
>> > at
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>> >
>> >
>> at
>>
>> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>> > at
>> >
>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>> >
>> >
>> at
>>
>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
>> > at
>> >
>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>> >
>> >
>> at
>>
>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
>> > at
>> >
>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>> >
>> >
>> at
>>
>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
>> > at
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>> >
>> >
>> at
>>
>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>> > at
>> >
>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>> >
>> >
>> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >
>> >
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> > at
>> >
>> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.execute(HTTPEngine4Impl.java:416)
>> >
>> >
>> at
>>
>> lucee.commons.net.http.httpclient4.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:252)
>> > at
>> >
>> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.get(HTTPEngine4Impl.java:112)
>> >
>> >
>> at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:86)
>> > at
>> > lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:108)
>> >
>> >
>> at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:58)
>>
>>
>> Same with this one:
>>
>> "Thread-27551" Id=27624 RUNNABLE (in native)
>>    java.lang.Thread.State: RUNNABLE
>>
>> But neither of the above threads making HTTP requests are
>> request-processing threads: they are independent and shouldn't block
>> incoming requests.
>>
>> You have 12 total request-processing threads: the ones named
>> "http-bio-8888-exec-[#]". Most of them aren't doing anything -- just
>> waiting on a new request to come in.
>>
>> You have an AJP connector defined, but it looks like it's not being used.
>>
>> Initially, you said that you used mod_proxy from httpd. Are you
>> expecting to use mod_proxy_http or mod_proxy_ajp?
>
>
> mod_proxy_http, as far as I know, but see below...
>
>
>> It certainly looks
>> like you are using the HTTP flavor of mod_proxy since your
>> file-uploads are going to the HTTP connector on port 8888 instead of
>> the AJP connector on port 8009. If you don't need the AJP connector,
>> you might want to disable it and get a small amount of resources back.
>>
>
> Okay, thanks.
>
> When you try to make a connection during these incidents, do you get
>> any errors on the httpd side?
>
>
> Unfortunately, at the time, I thought this would be easy to reproduce, so
> I didn't take enough notes (including the time of the problem on my load
> test box). Here are some errors that I think were just created by the app
> timing out requests. I don't think these were associated with the failed
> requests, so take these entries (from httpd's error_log) with a grain of
> salt:
> https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-errors-txt
>
> I'll capture this information next time I trigger a crash.
>
>
>> I know mod_jk will complain if it can't
>> make a connection or if there is a timeout... I suspect mod_proxy_http
>> will do the same.
>>
>> Can you post your <Connector> configuration from conf/server.xml?
>> Remember to remove any sensitive information that you might have in there.
>>
>
> I think this is what you're after:
>
>     <Connector port="8888" protocol="HTTP/1.1"
>                connectionTimeout="93000"
>                redirectPort="8443" />
>
> FWIW, I'm also noticing this:
>
>       <Host name="127.0.0.1"  appBase="webapps"
>             unpackWARs="false" autoDeploy="false">
>
>         <!-- Access log processes all example.
>              Documentation at: /docs/config/valve.html
>              Note: The pattern used is equivalent to using
> pattern="common" -->
>         <Valve className="org.apache.catalina.valves.AccessLogValve"
> directory="logs"
>                prefix="localhost_access_log." suffix=".txt"
>                pattern="%h %l %u %t &quot;%r&quot; %s %b"
> resolveHosts="false"/>
>
>         <!-- visit modcfml.org for details on mod_cfml configuration
> options -->
>         <Valve className="mod_cfml.core"
>                 loggingEnabled="false"
>                 waitForContext="3"
>                 maxContexts="200"
>                 timeBetweenContexts="30000"
>                 />
>       </Host>
>
>
> Also, what do your ProxyPass/ProxyPassReverse configuration directives
>> look like in httpd.conf?
>
>
> The apache config has:
>
> LoadModule proxy_module modules/mod_proxy.so
> LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
> LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
> LoadModule proxy_http_module modules/mod_proxy_http.so
> LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
> LoadModule proxy_connect_module modules/mod_proxy_connect.so
>
> And the virtual host has:
> https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-virtual-host
>
>
>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Comment: GPGTools - http://gpgtools.org
>>
>> iQIcBAEBCAAGBQJWHQhoAAoJEBzwKT+lPKRYoVwP/08YV4m1q3jXp+ixsPaPReEf
>> m9wnqPlWpX9tGjwVcsYvnqMPTJe8Ew+yzIFZ5NkPsmQk/eeKQFMYBsr0yDje4Nl0
>> Suk+OfNXhdL7RUn/PZg1Rd1xl7hcVeJmgOD6HRIcXAsnB+iTpCYpR+FtTnoGosia
>> PQ64AKEZR7KliP0bkM04NY4hHa+eA5vp2kGyGAAkA2g+sxUUXgiAVoUYMbLYwetv
>> CKqwlCFcG+JF9M2qi4n8vVwpsTqdLrInlO+yAiejZoQuHaSefnolIkFE0y+08bnz
>> Yv1nhNXwIOJAIvsl2BFoQIXXnpVh2ncILLIzhrcBBsazBy0RxccKsOJR0F6SKEWg
>> 3JLD6z/3CxtuW28KU2nFIQiKEQHzhZ+fV1NAp3XrHaQPncFGlNGPOUgy4Ggqxy3j
>> eAOkN1si3GFRm+ILELDO4ekKQCoAAdWs3MMpVTOKdL6mMO1OiYGPHzVTclgLloxc
>> o6FVgy81s6Hk9UJlu5enZJu9I+5tFxlD1gXrBmG7ryCOyiDn9pHmZ68EBpe3j+Zf
>> HPUtt7XP5eTVZp8vlVS3ebBHjrMkxUYGuBKYZF7prGfA5LE42nctdMslneiyZJJc
>> AOrcp32bgJFCcXmdBovD2UrJu0Evc4su3vV4JW9PuFSE1i3ntFLKuqL4FiArf1ls
>> +0KkHZDkU8C3MP3MCySA
>> =5Mun
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message