tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Thomas <ma...@apache.org>
Subject Re: (7.0.40) Seeing deadlock followed by maxThread usage using a Async Servlet AsyncContext.dispatch with BIO
Date Fri, 06 Sep 2013 13:24:30 GMT
On 06/09/2013 13:28, Daniel Mikusa wrote:
> On Sep 5, 2013, at 3:30 PM, Bob DeRemer <bob.deremer@thingworx.com> wrote:
> 
>> I’m trying to determine if there are any known issues using an Async servlet with
the default BIO connector in Tomcat 7.0.40 on windows server 2008.
>>  
>> PROBLEM:
>> We have a situation where we’re occasionally getting a deadlock (according to Java
Melody).  When this happens, it’s just a matter of time until all the threads in the connector’s
internal pool are exhausted because browser-based clients continue to make calls on this async
servlet – effectively polling for work. 
>>  
>> QUESTION:
>> I’m in the process of going through the stacks and code, but I’m not well versed
in the async servlet code (which is mostly from the Vysper project), so I’m trying to figure
out if the underlying AsyncContext.dispatch usage is a problem [at least when used with BIO].
  
>>  
>> Given the 3 stacks below, I’d greatly appreciate it if someone who understands
the Async servlet plumbing might be able to take a quick look and see if anything stands out
that could be the cause.  If you need any additional information, please let me know.
>>  
>> Thanks,
>> Bob
> 
> Is there any chance you could get a full thread dump while the problem is occurring?
 The thread dump below doesn't have any information about what locks are being held by the
blocked threads.  I think that might help to see why each thread is blocking.

+1.

What I would say is that there is no async code involved in the
http-bio-192.168.230.133-443-exec-1329 thread which makes me suspect the
vysper code.

The only locks I'd expect Tomcat to be holding are on the socket to
ensure that only one thread processes a connection at a time and I'm
pretty such there is no opportunity for deadlock there.

A closer look at BoshBackedSessionContext is probably in order.

Mark

> 
> Dan
> 
> 
>>  
>> DEADLOCKED STACK INFORMATION
>>  
>> Warning, the following threads are deadlocked : http-bio-192.168.230.133-443-exec-1329,
http-bio-192.168.230.133-443-exec-685, http-bio-192.168.230.133-443-exec-687
>>  
>> "http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED
>>        org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.java:556)
>>        org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:365)
>>        org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:848)
>>        org.apache.coyote.Request.action(Request.java:344)
>>        org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:222)
>>        org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:181)
>>        org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:175)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.write0(BoshBackedSessionContext.java:251)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession(BoshHandler.java:237)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(BoshHandler.java:228)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:192)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78)
>>        org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
>>        org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
>>        org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
>>        org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
>>        org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
>>        org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81)
>>        com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176)
>>        javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
>>        javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
>>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
>>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>        com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279)
>>        com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150)
>>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>        org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>        org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>        org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)
>>        org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>        org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>>        org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>>        org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
>>        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>        org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>        java.lang.Thread.run(Unknown Source)
>>  
>> "http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED
>>        org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.requestExpired(BoshBackedSessionContext.java:479)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.access$000(BoshBackedSessionContext.java:49)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.onTimeout(BoshBackedSessionContext.java:669)
>>        org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:45)
>>        org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:151)
>>        org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:293)
>>        org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(AbstractHttp11Processor.java:1568)
>>        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:583)
>>        org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>        java.lang.Thread.run(Unknown Source)
>>  
>> "http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED
>>        org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:160)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78)
>>        org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
>>        org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
>>        org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
>>        org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
>>        org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
>>        org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185)
>>        org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81)
>>        com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176)
>>        javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
>>        javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
>>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
>>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
>>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>        com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279)
>>        com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150)
>>        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>        org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>        org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>        org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)
>>        org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>        org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>>        org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>>        org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
>>        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>        org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>        java.lang.Thread.run(Unknown Source)
>>  
>>  
>> Bob DeRemer
>> Senior Director, Architecture and Development
>>  
>>
>> http://www.thingworx.com
>> Skype: bob.deremer.thingworx
>> O: 610.594.6200 x812
>> M: 717.881.3986
>>  
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message