tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Carr <andrewlanec...@gmail.com>
Subject Re: Parallel Classloader - Again
Date Tue, 27 Nov 2018 18:24:40 GMT
Tomcat 8.5.24
Java - (1.8.0-zing_16.12.3.0-b2-product-azlinuxM-X86_64 mixed mode [Mon Aug
13 05:18:16 PM UTC 2018]):

On Wed, Nov 21, 2018 at 6:49 AM Mark Thomas <markt@apache.org> wrote:

> And which version of Java?
>
> See also comments below.
>
> On 20/11/2018 22:16, Mark Thomas wrote:
> > Which version of Tomcat are you using?
> >
> > Mark
> >
> >
> > On 20/11/2018 21:02, Andrew Carr wrote:
> >> Wait, I think I might see what I did, let me try again.
> >>
> >> Found one Java-level deadlock:
> >> =============================
> >> "Thread-96":
> >>    waiting to lock monitor 0x3a0d6938 (object 0x80048462700, a
> >> java.util.concurrent.ConcurrentHashMap),
> >>    which is held by "Thread-29"
>
> The above thread can be ignored as it is blocked by the deadlock but
> isn't part of it.
>
> >> "Thread-29":
> >>    waiting to lock monitor 0x3a0d58b0 (object 0x1000bbc1dd20, a
> >> java.lang.Object),
> >>    which is held by "Event Manager Thread_QuartzSchedulerThread"
>
> I can see all of this in the provided stack traces.
>
>
> >> "Event Manager Thread_QuartzSchedulerThread":
> >>    waiting to lock monitor 0x3803a2c0 (object 0x800674b7830, a
> >> org.apache.catalina.loader.ParallelWebappClassLoader),
> >>    which is held by "Thread-29"
>
> There is no evidence in this stack trace that Thread 29 is holding that
> lock.
>
> There is no evidence in this stack trace that the Quartz thread is
> waiting for that lock.
>
> Hence my comment that the stack traces are not consistent with the
> reported deadlock.
>
> We really need to see where the lock on the ParallelWebappClassLoader
> object is being held. Maybe try getting a thread dump with a different
> tool?
>
> Until we can see where the locks are being held there isn't anything we
> can do.
>
> Generally, I'd recommend an update to the latest Tomcat point release
> and latest Java point release for whichever versions you are using just
> in case there is a JRE bug or similar contributing to this. I don't see
> anything related in the Tomcat changelog but I'd still always recommend
> updating.
>
> Mark
>
>
>
> >>
> >> "VM Thread" os_prio=0 tid=0x440000c00000 nid=0x1cbc
> >>
> >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000200000
> >> nid=0x1cb6
> >>
> >> "GC task thread#1 (GenPauselessGC)" os_prio=0 tid=0x440000400000
> >> nid=0x1cb7
> >>
> >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000600000
> >> nid=0x1cb8
> >>
> >> "VM Periodic Task Thread" os_prio=0 tid=0x440002e00000 nid=0x1ccd
> >>
> >> JNI global references: 98048
> >>
> >>
> >> Java stack information for the threads listed above:
> >> ===================================================
> >> "Thread-96":
> >> at
> >>
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
> >>
> >> - waiting to lock <0x0000080048462700> (a
> >> java.util.concurrent.ConcurrentHashMap)
> >> at
> >>
> org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
> >>
> >> at
> >>
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
> >>
> >> at
> >>
> com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.launchProvisioning(ClusterProvisioningThread.java:75)
> >>
> >> at
> >>
> com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.run(ClusterProvisioningThread.java:63)
> >>
> >> at java.lang.Thread.run(Thread.java:745)
> >> "Thread-29":
> >> at
> >>
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1152)
> >>
> >> - waiting to lock <0x00001000bbc1dd20> (a java.lang.Object)
> >> at
> >>
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
> >>
> >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:488)
> >> at
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.generateJobDetail(ScheduleEngine.java:982)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleAtFixedRate(ScheduleEngine.java:107)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleMinutely(ScheduleEngine.java:1316)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngineService.scheduleMinutely(ScheduleEngineService.java:493)
> >>
> >> - locked <0x00001000aa2752d8> (a
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.KQuartzEngineService)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:1057)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:876)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.tenantBasedSchedAndDeleteTask(EvtMgrStartUp.java:186)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.initialize(EvtMgrStartUp.java:138)
> >>
> >> - locked <0x0000080058b5c5e8> (a java.lang.Class)
> >> at
> >>
> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.startup(EvtMgrStartUp.java:73)
> >>
> >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>
> >> at java.lang.reflect.Method.invoke(Method.java:498)
> >> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
> >>
> >> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
> >>
> >> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> >>
> >> - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap)
> >> at
> >>
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
> >>
> >> at
> >>
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
> >>
> >> at
> >>
> com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startDeferredStartupServices(DeferredSpringServicesStarter.java:83)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startup(DeferredSpringServicesStarter.java:64)
> >>
> >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>
> >> at java.lang.reflect.Method.invoke(Method.java:498)
> >> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
> >>
> >> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
> >>
> >> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> >>
> >> - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap)
> >> at
> >>
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
> >>
> >> at
> >>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
> >>
> >> at
> >>
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
> >>
> >> at
> >>
> com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.startDeferredServices(WFCStartUp.java:1264)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1159)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1277)
> >>
> >> at
> >>
> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp$WFCInitThread.run(WFCStartUp.java:1429)
> >>
> >> "Event Manager Thread_QuartzSchedulerThread":
> >> at java.lang.ClassLoader.defineClass1(Native Method)
> >> at java.lang.ClassLoader.defineClass(ClassLoader.java:769)
> >> at
> >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> >> at
> >>
> org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283)
> >>
> >> - locked <0x00001000bbc1dd20> (a java.lang.Object)
> >> at
> >>
> org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811)
> >>
> >> at
> >>
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260)
> >>
> >> - locked <0x00001000bbc1dd20> (a java.lang.Object)
> >> at
> >>
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
> >>
> >> at
> >>
> org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59)
> >>
> >> at
> >>
> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99)
> >>
> >> at
> >>
> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138)
> >>
> >> at
> >>
> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
> >>
> >> at
> >>
> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385)
> >>
> >> at
> >>
> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818)
> >>
> >> at
> >>
> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759)
> >>
> >> at
> >>
> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757)
> >>
> >> at
> >>
> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799)
> >>
> >> at
> >>
> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756)
> >>
> >> at
> >>
> org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>

-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message