tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie Jackson <jamieja...@gmail.com>
Subject Re: Monitoring Connections
Date Fri, 16 Oct 2015 19:12:34 GMT
On Tue, Oct 13, 2015 at 9:34 AM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Jamie,
>
> On 10/9/15 10:03 AM, Jamie Jackson wrote:
> > Here's the stack trace dump:
> > https://gist.github.com/jamiejackson/ca2a49d2c8afac496067
> >
> > FWIW, I've been trying to come up with a reliable test case to
> > trigger the problem, but I haven't nailed it yet. I've suspected
> > that it's related to file (large or slow) HTTP file uploads, and
> > that's what I was running at the time, which helps to explain the
> > java.net.SocketInputStream.socketRead0 traces.
>
> Your server looks mostly idle. It also looks like it's running both
> Tomcat and Jetty in the same instance. What's going on?


I'm not sure yet. I'm asking around, and I'll let you know.


> > "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native)
> > java.lang.Thread.State: RUNNABLE at
> > java.net.SocketInputStream.socketRead0(Native Method) at
> > java.net.SocketInputStream.read(SocketInputStream.java:152) at
> > java.net.SocketInputStream.read(SocketInputStream.java:122) at
> >
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535)
> >
> >
> at
>
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
> > at
> >
> org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566)
> >
> >
> at
>
> org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137)
> > at
> >
> org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348)
> >
> >
> at org.apache.coyote.Request.doRead(Request.java:422)
> > at
> >
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
> >
> >
> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390)
> > at
> > org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304)
> >
> >
> at
>
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106)
> > at
> >
> com.intergral.fusionreactor.j2ee.filter.surrogate.FusionReactorServletRequestProxy$1.read(FusionReactorServletRequestProxy.java:123)
> >
> >
> at java.io.InputStream.read(InputStream.java:179)
> > at
> >
> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977)
> >
> >
> at
>
> org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)
> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) -
> > locked java.io.BufferedInputStream@6e81abfc at
> > java.io.FilterInputStream.read(FilterInputStream.java:107)
>
> Obviously, this thread is reading a file upload. Also these:
>
> "http-bio-8888-exec-21" Id=27309 RUNNABLE (in native)
>    java.lang.Thread.State: RUNNABLE
> "http-bio-8888-exec-26" Id=27314 RUNNABLE (in native)
>    java.lang.Thread.State: RUNNABLE
>
> This thread is making an outgoing HTTP request, which could be hanging:
>
> > "Thread-27550" Id=27623 RUNNABLE (in native)
> > java.lang.Thread.State: RUNNABLE at
> > java.net.SocketInputStream.socketRead0(Native Method) at
> > java.net.SocketInputStream.read(SocketInputStream.java:152) at
> > java.net.SocketInputStream.read(SocketInputStream.java:122) at
> > sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at
> > sun.security.ssl.InputRecord.read(InputRecord.java:480) at
> > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) -
> > locked java.lang.Object@659eed19 at
> > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
> >
> >
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
> > - locked sun.security.ssl.AppInputStream@7c98d0ab at
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
> >
> >
> at
>
> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> > at
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
> >
> >
> at
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> > at
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> >
> >
> at
>
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> > at
> >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
> >
> >
> at
>
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> > at
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
> >
> >
> at
>
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> > at
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> >
> >
> at
>
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
> > at
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
> >
> >
> at
>
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> > at
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> >
> >
> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> >
> at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.execute(HTTPEngine4Impl.java:416)
> >
> >
> at
>
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:252)
> > at
> >
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.get(HTTPEngine4Impl.java:112)
> >
> >
> at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:86)
> > at
> > lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:108)
> >
> >
> at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:58)
>
>
> Same with this one:
>
> "Thread-27551" Id=27624 RUNNABLE (in native)
>    java.lang.Thread.State: RUNNABLE
>
> But neither of the above threads making HTTP requests are
> request-processing threads: they are independent and shouldn't block
> incoming requests.
>
> You have 12 total request-processing threads: the ones named
> "http-bio-8888-exec-[#]". Most of them aren't doing anything -- just
> waiting on a new request to come in.
>
> You have an AJP connector defined, but it looks like it's not being used.
>
> Initially, you said that you used mod_proxy from httpd. Are you
> expecting to use mod_proxy_http or mod_proxy_ajp?


mod_proxy_http, as far as I know, but see below...


> It certainly looks
> like you are using the HTTP flavor of mod_proxy since your
> file-uploads are going to the HTTP connector on port 8888 instead of
> the AJP connector on port 8009. If you don't need the AJP connector,
> you might want to disable it and get a small amount of resources back.
>

Okay, thanks.

When you try to make a connection during these incidents, do you get
> any errors on the httpd side?


Unfortunately, at the time, I thought this would be easy to reproduce, so I
didn't take enough notes (including the time of the problem on my load test
box). Here are some errors that I think were just created by the app timing
out requests. I don't think these were associated with the failed requests,
so take these entries (from httpd's error_log) with a grain of salt:
https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-errors-txt

I'll capture this information next time I trigger a crash.


> I know mod_jk will complain if it can't
> make a connection or if there is a timeout... I suspect mod_proxy_http
> will do the same.
>
> Can you post your <Connector> configuration from conf/server.xml?
> Remember to remove any sensitive information that you might have in there.
>

I think this is what you're after:

    <Connector port="8888" protocol="HTTP/1.1"
               connectionTimeout="93000"
               redirectPort="8443" />

FWIW, I'm also noticing this:

      <Host name="127.0.0.1"  appBase="webapps"
            unpackWARs="false" autoDeploy="false">

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common"
-->
        <Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b"
resolveHosts="false"/>

        <!-- visit modcfml.org for details on mod_cfml configuration
options -->
        <Valve className="mod_cfml.core"
                loggingEnabled="false"
                waitForContext="3"
                maxContexts="200"
                timeBetweenContexts="30000"
                />
      </Host>


Also, what do your ProxyPass/ProxyPassReverse configuration directives
> look like in httpd.conf?


The apache config has:

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so

And the virtual host has:
https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-virtual-host


> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJWHQhoAAoJEBzwKT+lPKRYoVwP/08YV4m1q3jXp+ixsPaPReEf
> m9wnqPlWpX9tGjwVcsYvnqMPTJe8Ew+yzIFZ5NkPsmQk/eeKQFMYBsr0yDje4Nl0
> Suk+OfNXhdL7RUn/PZg1Rd1xl7hcVeJmgOD6HRIcXAsnB+iTpCYpR+FtTnoGosia
> PQ64AKEZR7KliP0bkM04NY4hHa+eA5vp2kGyGAAkA2g+sxUUXgiAVoUYMbLYwetv
> CKqwlCFcG+JF9M2qi4n8vVwpsTqdLrInlO+yAiejZoQuHaSefnolIkFE0y+08bnz
> Yv1nhNXwIOJAIvsl2BFoQIXXnpVh2ncILLIzhrcBBsazBy0RxccKsOJR0F6SKEWg
> 3JLD6z/3CxtuW28KU2nFIQiKEQHzhZ+fV1NAp3XrHaQPncFGlNGPOUgy4Ggqxy3j
> eAOkN1si3GFRm+ILELDO4ekKQCoAAdWs3MMpVTOKdL6mMO1OiYGPHzVTclgLloxc
> o6FVgy81s6Hk9UJlu5enZJu9I+5tFxlD1gXrBmG7ryCOyiDn9pHmZ68EBpe3j+Zf
> HPUtt7XP5eTVZp8vlVS3ebBHjrMkxUYGuBKYZF7prGfA5LE42nctdMslneiyZJJc
> AOrcp32bgJFCcXmdBovD2UrJu0Evc4su3vV4JW9PuFSE1i3ntFLKuqL4FiArf1ls
> +0KkHZDkU8C3MP3MCySA
> =5Mun
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message