Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
User-Agent: K-9 Mail for Android
In-Reply-To: 
 <AMXPR01MB0089ACE5A3F9B083E010B8BFC8B0@AMXPR01MB008.eurprd01.prod.exchangelabs.com>
References: 
 <AMXPR01MB0089ACE5A3F9B083E010B8BFC8B0@AMXPR01MB008.eurprd01.prod.exchangelabs.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain;
 charset=UTF-8
Subject: Re: Tomcat 8 Hung State
From: Mark Thomas <markt@apache.org>
Date: Thu, 17 Mar 2016 17:36:23 +0000
To: Tomcat Users List <users@tomcat.apache.org>
Message-ID: <083ECCAE-11E9-459E-9FA4-CF251B616B72@apache.org>

On 17 March 2016 16:13:28 GMT+00:00, Theo Sweeny <theo.sweeny@avios.com> wrote:
>Hello - yesterday Tomcat 8 went into a hung state, because an external
>API went offline.
>
>
>Eventually the external service was restored but Tomcat was unable to
>recovery from it.
>
>
>When a new connection is sent to Tomcat the connection hangs as no
>additional thread can be allocated to the new request.
>
>
>Here is an example of the log -
>
>
>17-Mar-2016 15:53:36.671 WARNING
>[ContainerBackgroundProcessor[StandardEngine[Catalina].StandardHost[localhost]]]
>org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected
>Thread "http-nio-10180-exec-40" (id=422) has been active for 33,717
>milliseconds (since 3/17/16 3:53 PM) to serve the same request for
>http://localhost:10180/retrieve-1.2/retrieve-prices and may be stuck
>(configured threshold for this StuckThreadDetectionValve is 30
>seconds). There is/are 1 thread(s) in total that are monitored by this
>Valve and may be stuck.
> java.lang.Throwable
>    at sun.misc.Unsafe.park(Native Method)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
>    at java.util.concurrent.FutureTask.get(FutureTask.java:187)
>at
>com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0(IlogPriceRepository.java:104)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$AjcClosure1.run(IlogPriceRepository.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.repository.product.price.IlogPriceRepository.getPrices(IlogPriceRepository.java:86)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$$FastClassBySpringCGLIB$$142c62f1.invoke(<generated>)
>at
>org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>at
>org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
>at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>at
>org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
>at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>at
>org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$$EnhancerBySpringCGLIB$$4fdb7af0.getPrices(<generated>)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices_aroundBody0(RealTimePriceApplicationService.java:48)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService$AjcClosure1.run(RealTimePriceApplicationService.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices(RealTimePriceApplicationService.java:47)
>at
>com.spring.application.service.product.price.GetPriceApplicationController.getPrices_aroundBody0(GetPriceApplicationController.java:74)
>at
>com.spring.application.service.product.price.GetPriceApplicationController$AjcClosure1.run(GetPriceApplicationController.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.service.product.price.GetPriceApplicationController.getPrices(GetPriceApplicationController.java:70)
>    at sun.reflect.GeneratedMethodAccessor1306.invoke(Unknown Source)
>at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    at java.lang.reflect.Method.invoke(Method.java:606)
>at
>org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
>at
>org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
>at
>org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
>at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777)
>at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706)
>at
>org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>at
>org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
>at
>org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
>at
>org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
>at
>org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
>at
>org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
>at
>org.springframework.boot.context.web.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
>at
>org.springframework.boot.context.web.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:88)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:101)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
>at
>org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
>at
>org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>at
>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
>at
>org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
>at
>org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
>at
>org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
>at
>org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
>at
>org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
>at
>org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
>at
>org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
>at
>org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
>at
>org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1517)
>at
>org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1474)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at
>org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>    at java.lang.Thread.run(Thread.java:745)
>
>
>Are there any ideas as to why Tomcat is unable to free up its threads?

That is an application problem. Nothing to do with Tomcat code. Once a thread enters application code Tomcat has no control over it.

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org