tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob DeRemer <bob.dere...@thingworx.com>
Subject RE: (7.0.40) Seeing deadlock followed by maxThread usage using a Async Servlet AsyncContext.dispatch with BIO
Date Fri, 06 Sep 2013 13:45:11 GMT


> -----Original Message-----
> From: Mark Thomas [mailto:markt@apache.org]
> Sent: Friday, September 06, 2013 9:25 AM
> To: users@tomcat.apache.org
> Subject: Re: (7.0.40) Seeing deadlock followed by maxThread usage using a
> Async Servlet AsyncContext.dispatch with BIO
> 
> On 06/09/2013 13:28, Daniel Mikusa wrote:
> > On Sep 5, 2013, at 3:30 PM, Bob DeRemer <bob.deremer@thingworx.com>
> wrote:
> >
> >> I'm trying to determine if there are any known issues using an Async servlet
> with the default BIO connector in Tomcat 7.0.40 on windows server 2008.
> >>
> >> PROBLEM:
> >> We have a situation where we're occasionally getting a deadlock (according
> to Java Melody).  When this happens, it's just a matter of time until all the
> threads in the connector's internal pool are exhausted because browser-based
> clients continue to make calls on this async servlet - effectively polling for
> work.
> >>
> >> QUESTION:
> >> I'm in the process of going through the stacks and code, but I'm not well
> versed in the async servlet code (which is mostly from the Vysper project), so
> I'm trying to figure out if the underlying AsyncContext.dispatch usage is a
> problem [at least when used with BIO].
> >>
> >> Given the 3 stacks below, I'd greatly appreciate it if someone who
> understands the Async servlet plumbing might be able to take a quick look and
> see if anything stands out that could be the cause.  If you need any additional
> information, please let me know.
> >>
> >> Thanks,
> >> Bob
> >
> > Is there any chance you could get a full thread dump while the problem is
> occurring?  The thread dump below doesn't have any information about what
> locks are being held by the blocked threads.  I think that might help to see why
> each thread is blocking.
> 
> +1.
> 
> What I would say is that there is no async code involved in the
> http-bio-192.168.230.133-443-exec-1329 thread which makes me suspect the
> vysper code.
> 
> The only locks I'd expect Tomcat to be holding are on the socket to ensure that
> only one thread processes a connection at a time and I'm pretty such there is
> no opportunity for deadlock there.
> 
> A closer look at BoshBackedSessionContext is probably in order.
> 
> Mark
> 

Thanks for the suggestions!  I have already found the problem, which (as you suspected), is
in the BoshBackedSessionContext.  There is a potential deadlock between the BoshBackedSessionContext
and the underlying JIoEndpoint.  It's happening because the Vypser code has 2 methods using
"synchronized":

 - 1 thd is doing an AsyncContext.dispatch from a synchronized method of BoshBackedSessionContext
which hangs waiting for the JIoEndpoint synchronized socket wrapper in processSocketAsync
- the other thd is the async timeout in JIoEndpoint firing back into the BoshBackedSessionContext;
he acquired the socket wrapper sync and is attempting to a different synchronized method on
the same BoshBackedSessionContext

So, I'm going to remove the synchronized usage from the BoshBackedSessionContext and replace
with a Lock/Condition using a tryLock with timeout and 1 retry.  I've used this approach before
and it's typically a lot more performant and provides better control over situations like
this.

Thx, bob

> >
> > Dan
> >
> >
> >>
> >> DEADLOCKED STACK INFORMATION
> >>
> >> Warning, the following threads are deadlocked :
> >> http-bio-192.168.230.133-443-exec-1329,
> >> http-bio-192.168.230.133-443-exec-685,
> >> http-bio-192.168.230.133-443-exec-687
> >>
> >> "http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED
> >>
> org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.java:55
> 6)
> >>
> org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:
> 365)
> >>
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Proce
> ssor.java:848)
> >>        org.apache.coyote.Request.action(Request.java:344)
> >>
> org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:22
> 2)
> >>
> org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:18
> 1)
> >>
> org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:17
> 5)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.write0(
> BoshBackedSessionContext.java:251)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession(Bos
> hHandler.java:237)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(BoshH
> andler.java:228)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.
> java:192)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanz
> a(BoshSaxContentHandler.java:90)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endEleme
> nt(BoshSaxContentHandler.java:78)
> >>
> org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
> >>        org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
> >>
> org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
> >>
> org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
> >>        org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
> >>
> org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultN
> onBlockingXMLReader.java:185)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder
> .java:81)
> >>
> com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(
> ThingworxBoshServlet.java:176)
> >>        javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
> >>        javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> >>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt
> erChain.java:305)
> >>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> java:210)
> >>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
> >>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
> >>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt
> erChain.java:243)
> >>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> java:210)
> >>
> com.thingworx.security.filter.AuthenticationFilter.propagateRequest(Authentica
> tionFilter.java:279)
> >>
> com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.j
> ava:150)
> >>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt
> erChain.java:243)
> >>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> java:210)
> >>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
> e.java:222)
> >>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j
> ava:123)
> >>
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase
> .java:581)
> >>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:17
> 1)
> >>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> >>
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
> >>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav
> a:118)
> >>
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> >>
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Proc
> essor.java:1008)
> >>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstra
> ctProtocol.java:589)
> >>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:3
> 12)
> >>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> >>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> >>        java.lang.Thread.run(Unknown Source)
> >>
> >> "http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.request
> Expired(BoshBackedSessionContext.java:479)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.access$
> 000(BoshBackedSessionContext.java:49)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.onTi
> meout(BoshBackedSessionContext.java:669)
> >>
> org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListener
> Wrapper.java:45)
> >>
> org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:151
> )
> >>
> org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.ja
> va:293)
> >>
> org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(AbstractHttp
> 11Processor.java:1568)
> >>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstra
> ctProtocol.java:583)
> >>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:3
> 12)
> >>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> >>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> >>        java.lang.Thread.run(Unknown Source)
> >>
> >> "http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.
> java:160)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanz
> a(BoshSaxContentHandler.java:90)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endEleme
> nt(BoshSaxContentHandler.java:78)
> >>
> org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480)
> >>        org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
> >>
> org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154)
> >>
> org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108)
> >>        org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
> >>
> org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultN
> onBlockingXMLReader.java:185)
> >>
> org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder
> .java:81)
> >>
> com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(
> ThingworxBoshServlet.java:176)
> >>        javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
> >>        javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> >>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt
> erChain.java:305)
> >>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> java:210)
> >>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
> >>        net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
> >>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt
> erChain.java:243)
> >>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> java:210)
> >>
> com.thingworx.security.filter.AuthenticationFilter.propagateRequest(Authentica
> tionFilter.java:279)
> >>
> com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.j
> ava:150)
> >>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt
> erChain.java:243)
> >>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> java:210)
> >>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
> e.java:222)
> >>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j
> ava:123)
> >>
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase
> .java:581)
> >>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:17
> 1)
> >>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> >>
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
> >>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav
> a:118)
> >>
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> >>
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Proc
> essor.java:1008)
> >>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstra
> ctProtocol.java:589)
> >>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:3
> 10)
> >>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> >>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> >>        java.lang.Thread.run(Unknown Source)
> >>
> >>
> >> Bob DeRemer
> >> Senior Director, Architecture and Development
> >>
> >>
> >> http://www.thingworx.com
> >> Skype: bob.deremer.thingworx
> >> O: 610.594.6200 x812
> >> M: 717.881.3986
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message