tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Mikusa <dmik...@gopivotal.com>
Subject Re: (7.0.40) Seeing deadlock followed by maxThread usage using a Async Servlet AsyncContext.dispatch with BIO
Date Fri, 06 Sep 2013 12:28:55 GMT
On Sep 5, 2013, at 3:30 PM, Bob DeRemer <bob.deremer@thingworx.com> wrote:

> I’m trying to determine if there are any known issues using an Async servlet with the
default BIO connector in Tomcat 7.0.40 on windows server 2008.
>  
> PROBLEM:
> We have a situation where we’re occasionally getting a deadlock (according to Java
Melody).  When this happens, it’s just a matter of time until all the threads in the connector’s
internal pool are exhausted because browser-based clients continue to make calls on this async
servlet – effectively polling for work. 
>  
> QUESTION:
> I’m in the process of going through the stacks and code, but I’m not well versed
in the async servlet code (which is mostly from the Vysper project), so I’m trying to figure
out if the underlying AsyncContext.dispatch usage is a problem [at least when used with BIO].
  
>  
> Given the 3 stacks below, I’d greatly appreciate it if someone who understands the
Async servlet plumbing might be able to take a quick look and see if anything stands out that
could be the cause.  If you need any additional information, please let me know.
>  
> Thanks,
> Bob

Is there any chance you could get a full thread dump while the problem is occurring?  The
thread dump below doesn't have any information about what locks are being held by the blocked
threads.  I think that might help to see why each thread is blocking.

Dan


>  
> DEADLOCKED STACK INFORMATION
>  
> Warning, the following threads are deadlocked : http-bio-192.168.230.133-443-exec-1329,
http-bio-192.168.230.133-443-exec-685, http-bio-192.168.230.133-443-exec-687
>  
> "http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED
>        org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.java:556)
>        org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:365)
>        org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:848)
>        org.apache.coyote.Request.action(Request.java:344)
>        org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:222)
>        org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:181)
>        org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:175)
>        org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.write0(BoshBackedSessionContext.java:251)
>        org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession(BoshHandler.java:237)
>        org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(BoshHandler.java:228)
>        org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:192)
>        org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90)
>        org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78)
>        org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
>        org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
>        org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
>        org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
>        org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
>        org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185)
>        org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81)
>        com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176)
>        javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
>        javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>        com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279)
>        com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150)
>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>        org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>        org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>        org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)
>        org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>        org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>        org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>        org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
>        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>        org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>        java.lang.Thread.run(Unknown Source)
>  
> "http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED
>        org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.requestExpired(BoshBackedSessionContext.java:479)
>        org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.access$000(BoshBackedSessionContext.java:49)
>        org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.onTimeout(BoshBackedSessionContext.java:669)
>        org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:45)
>        org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:151)
>        org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:293)
>        org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(AbstractHttp11Processor.java:1568)
>        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:583)
>        org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>        java.lang.Thread.run(Unknown Source)
>  
> "http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED
>        org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:160)
>        org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90)
>        org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78)
>        org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
>        org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
>        org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
>        org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
>        org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
>        org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185)
>        org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81)
>        com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176)
>        javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
>        javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>        com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279)
>        com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150)
>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>        org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>        org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>        org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)
>        org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>        org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>        org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>        org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
>        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>        org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>        java.lang.Thread.run(Unknown Source)
>  
>  
> Bob DeRemer
> Senior Director, Architecture and Development
>  
> 
> http://www.thingworx.com
> Skype: bob.deremer.thingworx
> O: 610.594.6200 x812
> M: 717.881.3986
>  


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message