tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Richard <charle...@thelearningbar.com>
Subject Re: Tomcat thread dump analysis
Date Wed, 08 May 2013 14:52:31 GMT
Chris,

Top-posting is a post after another one I'm assuming? And sorry if it
wasn't related to Tomcat, I was just excited to finally making a bit of
headway on this issue.

Here is a full thread dump, sorry in advance if this doesn't follow the
etiquette on posting thread dumps:

"TP-Processor396" daemon prio=10 tid=0x00002aff2ba9d000 nid=0x5a7b waiting
on condition [0x00002aff61e98000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002afecfb91da0> (a
java.util.concurrent.Semaphore$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
        at
com.tc.object.locks.LockStateNode$PendingLockHold.park(LockStateNode.java:179)
        at
com.tc.object.locks.ClientLockImpl.acquireQueued(ClientLockImpl.java:723)
        at
com.tc.object.locks.ClientLockImpl.acquireQueued(ClientLockImpl.java:701)
        at com.tc.object.locks.ClientLockImpl.lock(ClientLockImpl.java:52)
        at
com.tc.object.locks.ClientLockManagerImpl.lock(ClientLockManagerImpl.java:98)
        at com.tc.object.bytecode.ManagerImpl.lock(ManagerImpl.java:747)
        at
com.tc.object.bytecode.ManagerUtilInternal.beginLock(ManagerUtilInternal.java:33)
        at
org.terracotta.locking.strategy.LongLockStrategy.beginLock(LongLockStrategy.java:16)
        at
org.terracotta.locking.strategy.LongLockStrategy.beginLock(LongLockStrategy.java:7)
        at
com.terracotta.toolkit.collections.ConcurrentDistributedMapDso.beginLock(ConcurrentDistributedMapDso.java:828)
        at
com.terracotta.toolkit.collections.ConcurrentDistributedMapDso.get(ConcurrentDistributedMapDso.java:195)
        at
com.terracotta.toolkit.collections.ConcurrentDistributedMapDsoArray.get(ConcurrentDistributedMapDsoArray.java:175)
        at
org.terracotta.collections.ConcurrentDistributedMap.get(ConcurrentDistributedMap.java:190)
        at
org.terracotta.cache.TerracottaDistributedCache.getNonExpiredEntry(TerracottaDistributedCache.java:197)
        at
org.terracotta.cache.TerracottaDistributedCache.getNonExpiredEntryCoherent(TerracottaDistributedCache.java:131)
        at
org.terracotta.cache.TerracottaDistributedCache.containsKey(TerracottaDistributedCache.java:126)
        at
org.terracotta.modules.ehcache.store.ClusteredStore.internalContainsKey(ClusteredStore.java:476)
        at
org.terracotta.modules.ehcache.store.ClusteredStore.containsKey(ClusteredStore.java:456)
        at
org.terracotta.modules.ehcache.store.ClusteredStore.containsKeyInMemory(ClusteredStore.java:463)
        at net.sf.ehcache.Cache.searchInStoreWithStats(Cache.java:1742)
        at net.sf.ehcache.Cache.get(Cache.java:1405)
        at
net.sf.ehcache.hibernate.regions.EhcacheTransactionalDataRegion.get(EhcacheTransactionalDataRegion.java:91)
        at
net.sf.ehcache.hibernate.strategy.AbstractReadWriteEhcacheAccessStrategy.get(AbstractReadWriteEhcacheAccessStrategy.java:65)
        at
org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:524)
        at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:397)
        at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
        at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
        at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
        at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)
        at
org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)
        at org.hibernate.type.ManyToOneType.assemble(ManyToOneType.java:219)
        at org.hibernate.type.TypeFactory.assemble(TypeFactory.java:443)
        at
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:119)
        at
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:105)
        at
org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:587)
        at
org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:542)
        at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:397)
        at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
        at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
        at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
        at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)
        at
org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)
        at org.hibernate.type.ManyToOneType.assemble(ManyToOneType.java:219)
        at org.hibernate.type.TypeFactory.assemble(TypeFactory.java:443)
        at
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:119)
        at
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:105)
        at
org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:587)
        at
org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:542)
        at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:397)
        at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
        at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
        at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
        at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)
        at
org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)
        at org.hibernate.type.EntityType.resolve(EntityType.java:412)
        at
org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:139)
        at
org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:877)
        at org.hibernate.loader.Loader.doQuery(Loader.java:752)
        at
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
        at org.hibernate.loader.Loader.loadEntity(Loader.java:1885)
        at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:71)
        at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:65)
        at
org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3062)
        at
org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:434)
        at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:415)
        at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
        at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
        at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
        at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)
        at
org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)
        at org.hibernate.type.EntityType.resolve(EntityType.java:412)
        at
org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:139)
        at
org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:877)
        at org.hibernate.loader.Loader.doQuery(Loader.java:752)
        at
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
        at org.hibernate.loader.Loader.loadEntity(Loader.java:1885)
        at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:71)
        at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:65)
        at
org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3062)
        at
org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:434)
        at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:415)
        at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
        at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
        at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
        at org.hibernate.impl.SessionImpl.get(SessionImpl.java:843)
        at org.hibernate.impl.SessionImpl.get(SessionImpl.java:836)
        at
org.springframework.orm.hibernate3.HibernateTemplate$1.doInHibernate(HibernateTemplate.java:528)
        at
org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:419)
        at
org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
        at
org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:522)
        at
org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:516)
        at
com.thelearningbar.persistence.hibernate.SurveyParticipantDaoImpl.getSurveyParticipantById(SurveyParticipantDaoImpl.java:243)
        at
com.thelearningbar.service.impl.SurveyParticipantManagerImpl.getSurveyParticipantById(SurveyParticipantManagerImpl.java:306)
        at sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy20.getSurveyParticipantById(Unknown Source)
        at
com.thelearningbar.web.SurveyFlowController.handleRequest(SurveyFlowController.java:187)
        at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
        at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
        at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
        at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:523)
        at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:463)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)
        at
org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
        at
org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
        at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
        at
org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
        at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
        at
org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
        at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
        at
org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
        at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
        at
org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
        at
org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)
        at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.terracotta.modules.tomcat.tomcat_5_5.SessionValve55.tcInvoke(SessionValve55.java:99)
        at
org.terracotta.modules.tomcat.tomcat_5_5.SessionValve55.invoke(SessionValve55.java:86)
        at
org.terracotta.session.ModernTomcatSessionValve.invoke(ModernTomcatSessionValve.java:66)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

Cheers,
Charles


On Wed, May 8, 2013 at 9:42 AM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Charles,
>
> On 5/8/13 8:31 AM, Charles Richard wrote:
> > On Wed, May 8, 2013 at 9:27 AM, Daniel Mikusa
> > <dmikusa@gopivotal.com> wrote:
> >> On May 8, 2013, at 8:20 AM, Charles Richard wrote:
> >>
> >>> Hi,
> >>>
> >>> We have a weird issue on our site which some random trigger
> >>> event will backup all c3p0 connections until it hits the max
> >>> pool size.
> >>>
> >>> I have scripts that will do a softReset on the c3p0 connection
> >>> pool when they hit their max so help us manage the issue and to
> >>> also help me have time to hopefully get some decent thread
> >>> dumps to catch the underlying issue.
> >>>
> >>> The problem happened yesterday and I get a lot of these:
> >>>
> >>> "TP-Processor396" daemon prio=10 tid=0x00002aff2ba9d000
> >>> nid=0x5a7b
> >> waiting
> >>> on condition [0x00002aff61e98000] java.lang.Thread.State:
> >>> WAITING (parking) at sun.misc.Unsafe.park(Native Method) -
> >>> parking to wait for  <0x00002afecfb91da0> (a
> >>> java.util.concurrent.Semaphore$NonfairSync) at
> >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >>>
> >>
> at
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> >>>
> >>
> at
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> >>>
> >>
> at
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
> >>>
> >>
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
> >>> at
> >>>
> >>
> com.tc.object.locks.LockStateNode$PendingLockHold.park(LockStateNode.java:179)
> >>>
> >>
> at
> >>>
> com.tc.object.locks.ClientLockImpl.acquireQueued(ClientLockImpl.java:723)
> >>>
> >>>
> at
> >>>
> com.tc.object.locks.ClientLockImpl.acquireQueued(ClientLockImpl.java:701)
> >>>
> >>>
> at com.tc.object.locks.ClientLockImpl.lock(ClientLockImpl.java:52)
> >>> at
> >>>
> >>
> com.tc.object.locks.ClientLockManagerImpl.lock(ClientLockManagerImpl.java:98)
> >>>
> >>
> at com.tc.object.bytecode.ManagerImpl.lock(ManagerImpl.java:747)
> >>
> >>>
> >>> If I look at a stack before the issue happened, I see no
> >>> TP-Processor threads with the "parking to wait for". What can i
> >>> read into this?
>
> >> What in your application is using the "com.tc.object.locks"
> >> package? This is not used by Tomcat.
> >
> > We are using Terracotta which is a bit of a black box to me (setup
> > I've inherited). Terracotta helps us with the Tomcat sessions being
> >  "transportable" across front end servers.
>
> Please don't top-post: it's confusing and hard to follow.
>
> Unfortunately, I don't see any Tomcat code in here anywhere, so I
> can't really help (yet). Is that the whole stack trace? I find it
> difficult to believe there "bytecode.ManagerImpl" is a Runnable or
> extends Thread, especially because the "run" method isn't the lowest
> in the stack trace.
>
> Can you give us more information?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJRikhNAAoJEBzwKT+lPKRY53QP/1oz2eR+CRG+g/+2Oedh1UjE
> FckiVRQFkLsFSJBelGdQU+x0wBljgiSiOUgVaEQotpnaLO8pQCFmjMPcCsAZ2YiY
> xmnfHf2s89w/YGXo7HNB5X1okhpn+1pzjznmUvHGLGKO5+OUTdJvZ2stS6m6uEfg
> Vtmn6IQZywUyLKOcZ/Tl/UESirYjIaT5ky3Mvxbb/9JAyXICutJtxZQdZDG7U1du
> UU7BwmMQP/7ZiCiR9zUoarr2MT5FTgy1mpBGGI0OopmZilvo83UiyBcJJ/lN5miz
> PlE0BqisOeZIk5hAX+FqUZ1rTFvjpPNQjfN/9TRelhOs9fP9ZaTQ4PIkjwP8h2R7
> V2aqQmIrrZSrjS2SRsPx+iEnd6klwGpvlPoCyuWhikjKdv06nwZ7iDLWePw9lek7
> x8q6KeuE5KUy5w0EGN23zx54sh92eipjXj1uBBWBoA8cA6DZEXEPG+cGGoj/cR9+
> 9mhdOUbKSd9MQUR4H1b2+sErTZhj1KeBPxnBxeUU7FVoIn7J/x+DMXtYMU5OX6Kq
> jO0YTAXUMGAb6WPR2pz5bU2nvlev8JVVDwtjmV6l3P+czAqVQ7aUDc6aRfPbcM+D
> RJb4i64Ww41ZgbJv4YQ5SkxgdLIcjlzMu07pPXUwonlQI8NGvZsMN60eeKXnrrX5
> tnst+5jt/p/7mSULqAtR
> =ATll
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message