tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob DeRemer <bob.dere...@thingworx.com>
Subject (7.0.40) Seeing deadlock followed by maxThread usage using a Async Servlet AsyncContext.dispatch with BIO
Date Thu, 05 Sep 2013 19:30:48 GMT
I'm trying to determine if there are any known issues using an Async servlet with the default
BIO connector in Tomcat 7.0.40 on windows server 2008.

PROBLEM:
We have a situation where we're occasionally getting a deadlock (according to Java Melody).
 When this happens, it's just a matter of time until all the threads in the connector's internal
pool are exhausted because browser-based clients continue to make calls on this async servlet
- effectively polling for work.

QUESTION:
I'm in the process of going through the stacks and code, but I'm not well versed in the async
servlet code (which is mostly from the Vysper project), so I'm trying to figure out if the
underlying AsyncContext.dispatch usage is a problem [at least when used with BIO].

Given the 3 stacks below, I'd greatly appreciate it if someone who understands the Async servlet
plumbing might be able to take a quick look and see if anything stands out that could be the
cause.  If you need any additional information, please let me know.

Thanks,
Bob


DEADLOCKED STACK INFORMATION

Warning, the following threads are deadlocked : http-bio-192.168.230.133-443-exec-1329, http-bio-192.168.230.133-443-exec-685,
http-bio-192.168.230.133-443-exec-687

"http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED
       org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.java:556)
       org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:365)
       org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:848)
       org.apache.coyote.Request.action(Request.java:344)
       org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:222)
       org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:181)
       org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:175)
       org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.write0(BoshBackedSessionContext.java:251)
       org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession(BoshHandler.java:237)
       org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(BoshHandler.java:228)
       org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:192)
       org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90)
       org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78)
       org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
       org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
       org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
       org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
       org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
       org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185)
       org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81)
       com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
       org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
       net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
       net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
       org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
       com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279)
       com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150)
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
       org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
       org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
       org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
       org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)
       org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
       org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
       org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
       org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
       org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
       org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
       org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
       org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
       java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
       java.lang.Thread.run(Unknown Source)

"http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED
       org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.requestExpired(BoshBackedSessionContext.java:479)
       org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.access$000(BoshBackedSessionContext.java:49)
       org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.onTimeout(BoshBackedSessionContext.java:669)
       org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:45)
       org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:151)
       org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:293)
       org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(AbstractHttp11Processor.java:1568)
       org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:583)
       org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
       java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
       java.lang.Thread.run(Unknown Source)

"http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED
       org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:160)
       org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90)
       org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78)
       org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
       org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
       org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
       org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
       org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
       org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185)
       org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81)
       com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
       org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
       net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
       net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
       org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
       com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279)
       com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150)
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
       org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
       org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
       org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
       org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)
       org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
       org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
       org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
       org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
       org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
       org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
       org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
       org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
       java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
       java.lang.Thread.run(Unknown Source)


Bob DeRemer
Senior Director, Architecture and Development

[Description: Description: Description: Description: cid:image001.png@01CBE3DE.51A12030]
http://www.thingworx.com<http://www.thingworx.com/>
Skype: bob.deremer.thingworx
O: 610.594.6200 x812
M: 717.881.3986


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message