tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob DeRemer <bob.dere...@thingworx.com>
Subject RE: (7.0.40) Seeing deadlock followed by maxThread usage using a Async Servlet AsyncContext.dispatch with BIO
Date Fri, 06 Sep 2013 13:55:25 GMT


> -----Original Message-----
> From: Bob DeRemer [mailto:bob.deremer@thingworx.com]
> Sent: Friday, September 06, 2013 9:45 AM
> To: Tomcat Users List
> Subject: RE: (7.0.40) Seeing deadlock followed by maxThread usage using a
> Async Servlet AsyncContext.dispatch with BIO
> 
> 
> 
> > -----Original Message-----
> > From: Mark Thomas [mailto:markt@apache.org]
> > Sent: Friday, September 06, 2013 9:25 AM
> > To: users@tomcat.apache.org
> > Subject: Re: (7.0.40) Seeing deadlock followed by maxThread usage
> > using a Async Servlet AsyncContext.dispatch with BIO
> >
> > On 06/09/2013 13:28, Daniel Mikusa wrote:
> > > On Sep 5, 2013, at 3:30 PM, Bob DeRemer <bob.deremer@thingworx.com>
> > wrote:
> > >
> > >> I'm trying to determine if there are any known issues using an
> > >> Async servlet
> > with the default BIO connector in Tomcat 7.0.40 on windows server 2008.
> > >>
> > >> PROBLEM:
> > >> We have a situation where we're occasionally getting a deadlock
> > >> (according
> > to Java Melody).  When this happens, it's just a matter of time until
> > all the threads in the connector's internal pool are exhausted because
> > browser-based clients continue to make calls on this async servlet -
> > effectively polling for work.
> > >>
> > >> QUESTION:
> > >> I'm in the process of going through the stacks and code, but I'm
> > >> not well
> > versed in the async servlet code (which is mostly from the Vysper
> > project), so I'm trying to figure out if the underlying
> > AsyncContext.dispatch usage is a problem [at least when used with BIO].
> > >>
> > >> Given the 3 stacks below, I'd greatly appreciate it if someone who
> > understands the Async servlet plumbing might be able to take a quick
> > look and see if anything stands out that could be the cause.  If you
> > need any additional information, please let me know.
> > >>
> > >> Thanks,
> > >> Bob
> > >
> > > Is there any chance you could get a full thread dump while the
> > > problem is
> > occurring?  The thread dump below doesn't have any information about
> > what locks are being held by the blocked threads.  I think that might
> > help to see why each thread is blocking.
> >
> > +1.
> >
> > What I would say is that there is no async code involved in the
> > http-bio-192.168.230.133-443-exec-1329 thread which makes me suspect
> > the vysper code.
> >
> > The only locks I'd expect Tomcat to be holding are on the socket to
> > ensure that only one thread processes a connection at a time and I'm
> > pretty such there is no opportunity for deadlock there.
> >
> > A closer look at BoshBackedSessionContext is probably in order.
> >
> > Mark
> >
> 
> Thanks for the suggestions!  I have already found the problem, which (as you
> suspected), is in the BoshBackedSessionContext.  There is a potential deadlock
> between the BoshBackedSessionContext and the underlying JIoEndpoint.  It's
> happening because the Vypser code has 2 methods using "synchronized":
> 
>  - 1 thd is doing an AsyncContext.dispatch from a synchronized method of
> BoshBackedSessionContext which hangs waiting for the JIoEndpoint
> synchronized socket wrapper in processSocketAsync
> - the other thd is the async timeout in JIoEndpoint firing back into the
> BoshBackedSessionContext; he acquired the socket wrapper sync and is
> attempting to a different synchronized method on the same
> BoshBackedSessionContext
> 
> So, I'm going to remove the synchronized usage from the
> BoshBackedSessionContext and replace with a Lock/Condition using a tryLock
> with timeout and 1 retry.  I've used this approach before and it's typically a lot
> more performant and provides better control over situations like this.
> 
> Thx, bob

CORRECTION: only a Lock is needed for this fix, no Condition; so accustomed to using them
together

> 
> > >
> > > Dan
> > >
> > >
> > >>
> > >> DEADLOCKED STACK INFORMATION
> > >>
> > >> Warning, the following threads are deadlocked :
> > >> http-bio-192.168.230.133-443-exec-1329,
> > >> http-bio-192.168.230.133-443-exec-685,
> > >> http-bio-192.168.230.133-443-exec-687
> > >>
> > >> "http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED
> > >>
> > org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.
> > java:55
> > 6)
> > >>
> >
> org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:
> > 365)
> > >>
> > org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11
> > Proce
> > ssor.java:848)
> > >>        org.apache.coyote.Request.action(Request.java:344)
> > >>
> > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.ja
> > va:22
> > 2)
> > >>
> > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.ja
> > va:18
> > 1)
> > >>
> > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.ja
> > va:17
> > 5)
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.writ
> > e0(
> > BoshBackedSessionContext.java:251)
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession(
> > Bos
> > hHandler.java:237)
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(Bo
> > shH
> > andler.java:228)
> > >>
> >
> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.
> > java:192)
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitSta
> > nz
> > a(BoshSaxContentHandler.java:90)
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElem
> > e
> > nt(BoshSaxContentHandler.java:78)
> > >>
> > org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480
> > )
> > >>
> > >> org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
> > >>
> > org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154
> > )
> > >>
> > org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:10
> > 8)
> > >>
> > >> org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
> > >>
> > org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(Defau
> > ltN
> > onBlockingXMLReader.java:185)
> > >>
> >
> org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecode
> > r
> > .java:81)
> > >>
> > com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPo
> > st(
> > ThingworxBoshServlet.java:176)
> > >>        javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
> > >>        javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> > >>
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> > cationFilt
> > erChain.java:305)
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> > java:210)
> > >>
> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
> > >>
> > >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java
> > >> :179)
> > >>
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> > cationFilt
> > erChain.java:243)
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> > java:210)
> > >>
> > com.thingworx.security.filter.AuthenticationFilter.propagateRequest(Au
> > thentica
> > tionFilter.java:279)
> > >>
> > com.thingworx.security.filter.AuthenticationFilter.doFilter(Authentica
> > tionFilter.j
> > ava:150)
> > >>
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> > cationFilt
> > erChain.java:243)
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> > java:210)
> > >>
> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
> > lv
> > e.java:222)
> > >>
> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
> > lve.j
> > ava:123)
> > >>
> > org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
> > orBase
> > .java:581)
> > >>
> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
> > va:17
> > 1)
> > >>
> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
> > va:99)
> > >>
> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:9
> > 53)
> > >>
> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
> > e.jav
> > a:118)
> > >>
> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
> > :408)
> > >>
> > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp1
> > 1Proc
> > essor.java:1008)
> > >>
> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(A
> > bstra
> > ctProtocol.java:589)
> > >>
> > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint
> > .java:3
> > 12)
> > >>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > >>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> > >>        java.lang.Thread.run(Unknown Source)
> > >>
> > >> "http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.requ
> > est
> > Expired(BoshBackedSessionContext.java:479)
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.acce
> > ss$
> > 000(BoshBackedSessionContext.java:49)
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.on
> > Ti
> > meout(BoshBackedSessionContext.java:669)
> > >>
> > org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListe
> > ner
> > Wrapper.java:45)
> > >>
> > org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.jav
> > a:151
> > )
> > >>
> > org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapte
> > r.ja
> > va:293)
> > >>
> > org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(Abstrac
> > tHttp
> > 11Processor.java:1568)
> > >>
> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(A
> > bstra
> > ctProtocol.java:583)
> > >>
> > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint
> > .java:3
> > 12)
> > >>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > >>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> > >>        java.lang.Thread.run(Unknown Source)
> > >>
> > >> "http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED
> > >>
> >
> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.
> > java:160)
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitSta
> > nz
> > a(BoshSaxContentHandler.java:90)
> > >>
> > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElem
> > e
> > nt(BoshSaxContentHandler.java:78)
> > >>
> > org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480
> > )
> > >>
> > >> org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227)
> > >>
> > org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154
> > )
> > >>
> > org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:10
> > 8)
> > >>
> > >> org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121)
> > >>
> > org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(Defau
> > ltN
> > onBlockingXMLReader.java:185)
> > >>
> >
> org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecode
> > r
> > .java:81)
> > >>
> > com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPo
> > st(
> > ThingworxBoshServlet.java:176)
> > >>        javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
> > >>        javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> > >>
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> > cationFilt
> > erChain.java:305)
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> > java:210)
> > >>
> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
> > >>
> > >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java
> > >> :179)
> > >>
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> > cationFilt
> > erChain.java:243)
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> > java:210)
> > >>
> > com.thingworx.security.filter.AuthenticationFilter.propagateRequest(Au
> > thentica
> > tionFilter.java:279)
> > >>
> > com.thingworx.security.filter.AuthenticationFilter.doFilter(Authentica
> > tionFilter.j
> > ava:150)
> > >>
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> > cationFilt
> > erChain.java:243)
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.
> > java:210)
> > >>
> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
> > lv
> > e.java:222)
> > >>
> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
> > lve.j
> > ava:123)
> > >>
> > org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
> > orBase
> > .java:581)
> > >>
> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
> > va:17
> > 1)
> > >>
> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
> > va:99)
> > >>
> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:9
> > 53)
> > >>
> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
> > e.jav
> > a:118)
> > >>
> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
> > :408)
> > >>
> > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp1
> > 1Proc
> > essor.java:1008)
> > >>
> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(A
> > bstra
> > ctProtocol.java:589)
> > >>
> > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint
> > .java:3
> > 10)
> > >>        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > >>        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> > >>        java.lang.Thread.run(Unknown Source)
> > >>
> > >>
> > >> Bob DeRemer
> > >> Senior Director, Architecture and Development
> > >>
> > >>
> > >> http://www.thingworx.com
> > >> Skype: bob.deremer.thingworx
> > >> O: 610.594.6200 x812
> > >> M: 717.881.3986
> > >>
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > For additional commands, e-mail: users-help@tomcat.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message