tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Theo Sweeny <theo.swe...@avios.com>
Subject Re: Tomcat 8 Hung State
Date Fri, 18 Mar 2016 15:06:40 GMT
Hi Mark,

________________________________________
From: Mark Thomas <markt@apache.org>
Sent: 17 March 2016 17:36
To: Tomcat Users List
Subject: Re: Tomcat 8 Hung State

On 17 March 2016 16:13:28 GMT+00:00, Theo Sweeny <theo.sweeny@avios.com> wrote:
>Hello - yesterday Tomcat 8 went into a hung state, because an external
>API went offline.
>
>
>Eventually the external service was restored but Tomcat was unable to
>recovery from it.
>
>
>When a new connection is sent to Tomcat the connection hangs as no
>additional thread can be allocated to the new request.
>
>
>Here is an example of the log -
>
>
>17-Mar-2016 15:53:36.671 WARNING
>[ContainerBackgroundProcessor[StandardEngine[Catalina].StandardHost[localhost]]]
>org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected
>Thread "http-nio-10180-exec-40" (id=422) has been active for 33,717
>milliseconds (since 3/17/16 3:53 PM) to serve the same request for
>http://localhost:10180/retrieve-1.2/retrieve-prices and may be stuck
>(configured threshold for this StuckThreadDetectionValve is 30
>seconds). There is/are 1 thread(s) in total that are monitored by this
>Valve and may be stuck.
> java.lang.Throwable
>    at sun.misc.Unsafe.park(Native Method)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
>    at java.util.concurrent.FutureTask.get(FutureTask.java:187)
>at
>com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0(IlogPriceRepository.java:104)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$AjcClosure1.run(IlogPriceRepository.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.repository.product.price.IlogPriceRepository.getPrices(IlogPriceRepository.java:86)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$$FastClassBySpringCGLIB$$142c62f1.invoke(<generated>)
>at
>org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>at
>org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
>at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>at
>org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
>at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>at
>org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$$EnhancerBySpringCGLIB$$4fdb7af0.getPrices(<generated>)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices_aroundBody0(RealTimePriceApplicationService.java:48)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService$AjcClosure1.run(RealTimePriceApplicationService.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices(RealTimePriceApplicationService.java:47)
>at
>com.spring.application.service.product.price.GetPriceApplicationController.getPrices_aroundBody0(GetPriceApplicationController.java:74)
>at
>com.spring.application.service.product.price.GetPriceApplicationController$AjcClosure1.run(GetPriceApplicationController.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.service.product.price.GetPriceApplicationController.getPrices(GetPriceApplicationController.java:70)
>    at sun.reflect.GeneratedMethodAccessor1306.invoke(Unknown Source)
>at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    at java.lang.reflect.Method.invoke(Method.java:606)
>at
>org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
>at
>org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
>at
>org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
>at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777)
>at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706)
>at
>org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>at
>org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
>at
>org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
>at
>org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
>at
>org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
>at
>org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
>at
>org.springframework.boot.context.web.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
>at
>org.springframework.boot.context.web.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:88)
>at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>at
>org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:101)
>at
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>at
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>at
>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
>at
>org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
>at
>org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>at
>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
>at
>org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
>at
>org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
>at
>org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
>at
>org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
>at
>org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
>at
>org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
>at
>org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
>at
>org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
>at
>org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1517)
>at
>org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1474)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at
>org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>    at java.lang.Thread.run(Thread.java:745)
>
>
>Are there any ideas as to why Tomcat is unable to free up its threads?

That is an application problem. Nothing to do with Tomcat code. Once a thread enters application
code Tomcat has no control over it.

Mark


Thanks for the reply - the developers are pushing back as they believe Tomcat should kill
off any rogue webapps after a certain timeframe as this is how they have it done in Weblogic.When
I do an Strace on a broken webapp, the following can be seen -

11297 11:39:23.023838 write(105, "2016-03-18 11:39:23,021 138066033 [http-nio-10080-exec-27]
ERROR com.spring.business.service.security.ldap.filter.CustomAuthFilter  - Exception thrown
in doFilterInternal: {}\norg.apache.catalina.connector.ClientAbortException: java.io.IOException:
Broken pipe\n\tat org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
~[catalina.jar:8.0.21]\n\tat org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
~[tomcat-util.jar:8.0.21]\n\tat org.apache.catalina.connector."..., 17236 <unfinished ...>

Is this due to Tomcat's inability to recover from a lost datasource?

Regards,

Theo
Avios Group (AGL) Ltd is a limited company registered in England (registered number 2260073
and VAT number 512566754) whose registered address is Astral Towers, Betts Way, London Road,
Crawley, West Sussex RH10 9XY . Avios Group (AGL) Limited is part of the IAG group of companies
This email and any files transmitted with it are confidential and intended solely for the
use of the individual or entity to whom they are addressed. If you have received this email
in error please notify the system manager.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message