tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rizwan Merchant <rizwan.merch...@sysconet.ca>
Subject Re: jvm thread dump
Date Tue, 10 Oct 2006 16:21:36 GMT

Thanks for the response Chris.
Actually, we had a bit of a situation yesterday, it seems the threads 
were all locked up and/or the db connection pool was not handing out 
connections. I am not sure what happened and have been monitoring the 
threads ever since.

Below are some threads from the dump when the above problem happened. I 
have no idea what caused the problem, I do know that the mail server I 
was trying to connect to was flaky, that would explain thread 
"http-443-Processor75" but I have no clue why the dbcp connection 
pooling failed.

Chris, are you familiar with Spring at all? Would you be able to answer 
a few questions for me on this subject (we can take the topic offline as 
it is not relevant here).

Thanks for your help. Appreciate it.

-Riz.

"http-443-Processor75" daemon prio=1 tid=0x09fbc620 nid=0x330a runnable 
[0x07d76000..0x07d790c0]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:79)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
    - locked <0x5cd7e218> (a java.io.BufferedInputStream)
    at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:57)
    at 
com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:1327)
    at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1191)
    at 
com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:311)
    at javax.mail.Service.connect(Service.java:233)
    at 
org.springframework.mail.javamail.JavaMailSenderImpl.doSend(JavaMailSenderImpl.java:373)
    at 
org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:326)
    at 
org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:322)
    at 
com.cwsi.eshipper.service.impl.EmailManagerImpl.sendMessageWithAttachments(EmailManagerImpl.java:209)
    at 
com.cwsi.eshipper.service.impl.EmailManagerImpl.onChargeCard(EmailManagerImpl.java:277)
    at 
com.cwsi.eshipper.service.impl.CCTransactionManagerImpl.emailReceipt(CCTransactionManagerImpl.java:435)
    at 
com.cwsi.eshipper.service.impl.CCTransactionManagerImpl.processPostAuth(CCTransactionManagerImpl.java:246)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:292)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:155)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:122)
    at 
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
    at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174)
    at $Proxy10.processPostAuth(Unknown Source)
    at 
com.cwsi.eshipper.service.impl.CarrierServiceManagerImpl.shipOrder(CarrierServiceManagerImpl.java:262)
    at sun.reflect.GeneratedMethodAccessor462.invoke(Unknown Source)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:292)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:155)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:122)
    at 
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
    at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174)
    at $Proxy19.shipOrder(Unknown Source)
    at 
com.cwsi.eshipper.web.OrderManagerAction.save_internal(OrderManagerAction.java:456)
    at 
com.cwsi.eshipper.web.OrderManagerAction.save(OrderManagerAction.java:1439)
    at sun.reflect.GeneratedMethodAccessor2710.invoke(Unknown Source)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at 
org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:274)
    at 
org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:194)
    at 
org.springframework.web.struts.DelegatingActionProxy.execute(DelegatingActionProxy.java:106)
    at 
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
    at 
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
    at 
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
    at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
org.displaytag.filter.ResponseOverrideFilter.doFilter(ResponseOverrideFilter.java:125)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:292)
    at 
net.sf.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:108)
    at 
net.sf.acegisecurity.intercept.web.SecurityEnforcementFilter.doFilter(SecurityEnforcementFilter.java:197)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:50)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:324)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:324)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:220)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:173)
    at 
net.sf.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:120)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142)
    at 
com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:172)
    at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
    at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
    at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
    at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
    at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
    at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
    at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
    at 
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
    at 
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
    at 
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)

"http-443-Processor74" daemon prio=1 tid=0x09fbb720 nid=0x3309 in 
Object.wait() [0x07cf7000..0x07cf8040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x55ed65c0> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x55ed65c0> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)

"http-443-Processor73" daemon prio=1 tid=0xb2ac23d0 nid=0x3308 in 
Object.wait() [0x07b96000..0x07b97fc0]
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:474)
    at 
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:748)
    - locked <0x539f5a70> (a org.apache.commons.pool.impl.GenericObjectPool)
    at 
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
    at 
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
    at 
org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:80)
    at 
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:417)
    at 
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
    at 
org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:40)
    at $Proxy18.isReadOnly(Unknown Source)
    at 
org.springframework.jdbc.datasource.DataSourceUtils.resetConnectionAfterTransaction(DataSourceUtils.java:186)
    at 
org.springframework.orm.hibernate3.HibernateTransactionManager.doCleanupAfterCompletion(HibernateTransactionManager.java:551)
    at 
org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:749)
    at 
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:536)
    at 
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:468)
    at 
org.springframework.transaction.interceptor.TransactionAspectSupport.doCommitTransactionAfterReturning(TransactionAspectSupport.java:258)
    at 
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
    at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174)
    at $Proxy15.search(Unknown Source)
    at 
com.cwsi.eshipper.web.OrderManagerAction.perform_search(OrderManagerAction.java:568)
    at sun.reflect.GeneratedMethodAccessor2674.invoke(Unknown Source)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at 
org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:274)
    at 
org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:194)
    at 
org.springframework.web.struts.DelegatingActionProxy.execute(DelegatingActionProxy.java:106)
    at 
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
    at 
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
    at 
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
    at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
org.displaytag.filter.ResponseOverrideFilter.doFilter(ResponseOverrideFilter.java:125)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:292)
    at 
net.sf.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:108)
    at 
net.sf.acegisecurity.intercept.web.SecurityEnforcementFilter.doFilter(SecurityEnforcementFilter.java:197)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:50)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:324)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:324)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:220)
    at 
net.sf.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:303)
    at 
net.sf.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:173)
    at 
net.sf.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:120)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142)
    at 
com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:172)
    at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
    at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
    at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
    at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
    at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
    at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
    at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
    at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
    at 
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
    at 
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
    at 
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)


Christopher Schultz wrote:
> Riz,
>
>   
>> When I perform a thread dump I am seeing some threads are locked on a
>> socketRead0 (like http-443-Processor49 in the dump below).
>>     
>
> Let's take a look:
>
>   
>> "http-443-Processor49" daemon prio=1 tid=0xb208fe80 nid=0x40c5 runnable
>> [0x06fdd000..0x06fddf40]
>>        at java.net.SocketInputStream.socketRead0(Native Method)
>>        at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>     
>
> What's wrong with that? Your thread is runnable (good), and it's reading
> data from an InputStream. Lots of the time, your request processor
> threads will be reading from sockets because they are accepting request
> information from the client. Several of your threads are in this state.
>
>   
>> Can someone PLEASE help me understand what this means? Also, if you can
>> see any other threads that seem to be a threat, please feel free to let
>> me know...
>>     
>
> I don't see any "threatening"-looking threads in there at all.
>
>   
>> "http-443-Processor50" daemon prio=1 tid=0xb2090c40 nid=0x40c6 in
>> Object.wait() [0x0705e000..0x0705efc0]
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x54f20ee8> (a
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
>>     
>
> Note that this request processor is blocking, waiting for a request to
> come in in order to be handled. This thread is basically idle. Many of
> your threads are in this state.
>
> Are you posting this question in response to any kind of observed
> problem, or are you just freaking out at the number of threads that are
> running in your JVM?
>
> -chris
>
>
>   

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message