tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Carr <andrewlanec...@gmail.com>
Subject Re: Parallel Classloader - Again
Date Tue, 27 Nov 2018 18:46:14 GMT
Mark,

Thanks for the other advice.  I have recommended upgrading as a possible
solution.  I do see the inconsistencies you are referring to, and I will
try to generate a thread dump with the issue happening using a different
tool, but the issue is intermittent.  Thank you much for the help, I'll msg
when I know more.

-Andrew

On Tue, Nov 27, 2018 at 1:24 PM Andrew Carr <andrewlanecarr@gmail.com>
wrote:

> Tomcat 8.5.24
> Java - (1.8.0-zing_16.12.3.0-b2-product-azlinuxM-X86_64 mixed mode [Mon
> Aug 13 05:18:16 PM UTC 2018]):
>
> On Wed, Nov 21, 2018 at 6:49 AM Mark Thomas <markt@apache.org> wrote:
>
>> And which version of Java?
>>
>> See also comments below.
>>
>> On 20/11/2018 22:16, Mark Thomas wrote:
>> > Which version of Tomcat are you using?
>> >
>> > Mark
>> >
>> >
>> > On 20/11/2018 21:02, Andrew Carr wrote:
>> >> Wait, I think I might see what I did, let me try again.
>> >>
>> >> Found one Java-level deadlock:
>> >> =============================
>> >> "Thread-96":
>> >>    waiting to lock monitor 0x3a0d6938 (object 0x80048462700, a
>> >> java.util.concurrent.ConcurrentHashMap),
>> >>    which is held by "Thread-29"
>>
>> The above thread can be ignored as it is blocked by the deadlock but
>> isn't part of it.
>>
>> >> "Thread-29":
>> >>    waiting to lock monitor 0x3a0d58b0 (object 0x1000bbc1dd20, a
>> >> java.lang.Object),
>> >>    which is held by "Event Manager Thread_QuartzSchedulerThread"
>>
>> I can see all of this in the provided stack traces.
>>
>>
>> >> "Event Manager Thread_QuartzSchedulerThread":
>> >>    waiting to lock monitor 0x3803a2c0 (object 0x800674b7830, a
>> >> org.apache.catalina.loader.ParallelWebappClassLoader),
>> >>    which is held by "Thread-29"
>>
>> There is no evidence in this stack trace that Thread 29 is holding that
>> lock.
>>
>> There is no evidence in this stack trace that the Quartz thread is
>> waiting for that lock.
>>
>> Hence my comment that the stack traces are not consistent with the
>> reported deadlock.
>>
>> We really need to see where the lock on the ParallelWebappClassLoader
>> object is being held. Maybe try getting a thread dump with a different
>> tool?
>>
>> Until we can see where the locks are being held there isn't anything we
>> can do.
>>
>> Generally, I'd recommend an update to the latest Tomcat point release
>> and latest Java point release for whichever versions you are using just
>> in case there is a JRE bug or similar contributing to this. I don't see
>> anything related in the Tomcat changelog but I'd still always recommend
>> updating.
>>
>> Mark
>>
>>
>>
>> >>
>> >> "VM Thread" os_prio=0 tid=0x440000c00000 nid=0x1cbc
>> >>
>> >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000200000
>> >> nid=0x1cb6
>> >>
>> >> "GC task thread#1 (GenPauselessGC)" os_prio=0 tid=0x440000400000
>> >> nid=0x1cb7
>> >>
>> >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000600000
>> >> nid=0x1cb8
>> >>
>> >> "VM Periodic Task Thread" os_prio=0 tid=0x440002e00000 nid=0x1ccd
>> >>
>> >> JNI global references: 98048
>> >>
>> >>
>> >> Java stack information for the threads listed above:
>> >> ===================================================
>> >> "Thread-96":
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
>> >>
>> >> - waiting to lock <0x0000080048462700> (a
>> >> java.util.concurrent.ConcurrentHashMap)
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
>> >>
>> >> at
>> >>
>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.launchProvisioning(ClusterProvisioningThread.java:75)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.run(ClusterProvisioningThread.java:63)
>> >>
>> >> at java.lang.Thread.run(Thread.java:745)
>> >> "Thread-29":
>> >> at
>> >>
>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1152)
>> >>
>> >> - waiting to lock <0x00001000bbc1dd20> (a java.lang.Object)
>> >> at
>> >>
>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
>> >>
>> >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:488)
>> >> at
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.generateJobDetail(ScheduleEngine.java:982)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleAtFixedRate(ScheduleEngine.java:107)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleMinutely(ScheduleEngine.java:1316)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngineService.scheduleMinutely(ScheduleEngineService.java:493)
>> >>
>> >> - locked <0x00001000aa2752d8> (a
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.KQuartzEngineService)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:1057)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:876)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.tenantBasedSchedAndDeleteTask(EvtMgrStartUp.java:186)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.initialize(EvtMgrStartUp.java:138)
>> >>
>> >> - locked <0x0000080058b5c5e8> (a java.lang.Class)
>> >> at
>> >>
>> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.startup(EvtMgrStartUp.java:73)
>> >>
>> >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
>> >> at
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>
>> >> at java.lang.reflect.Method.invoke(Method.java:498)
>> >> at
>> >>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> >>
>> >> - locked <0x0000080048462700> (a
>> java.util.concurrent.ConcurrentHashMap)
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
>> >>
>> >> at
>> >>
>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startDeferredStartupServices(DeferredSpringServicesStarter.java:83)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startup(DeferredSpringServicesStarter.java:64)
>> >>
>> >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
>> >> at
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>
>> >> at java.lang.reflect.Method.invoke(Method.java:498)
>> >> at
>> >>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> >>
>> >> - locked <0x0000080048462700> (a
>> java.util.concurrent.ConcurrentHashMap)
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
>> >>
>> >> at
>> >>
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
>> >>
>> >> at
>> >>
>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.startDeferredServices(WFCStartUp.java:1264)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1159)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1277)
>> >>
>> >> at
>> >>
>> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp$WFCInitThread.run(WFCStartUp.java:1429)
>> >>
>> >> "Event Manager Thread_QuartzSchedulerThread":
>> >> at java.lang.ClassLoader.defineClass1(Native Method)
>> >> at java.lang.ClassLoader.defineClass(ClassLoader.java:769)
>> >> at
>> >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> >> at
>> >>
>> org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283)
>> >>
>> >> - locked <0x00001000bbc1dd20> (a java.lang.Object)
>> >> at
>> >>
>> org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811)
>> >>
>> >> at
>> >>
>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260)
>> >>
>> >> - locked <0x00001000bbc1dd20> (a java.lang.Object)
>> >> at
>> >>
>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
>> >>
>> >> at
>> >>
>> org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59)
>> >>
>> >> at
>> >>
>> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99)
>> >>
>> >> at
>> >>
>> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138)
>> >>
>> >> at
>> >>
>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
>> >>
>> >> at
>> >>
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385)
>> >>
>> >> at
>> >>
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818)
>> >>
>> >> at
>> >>
>> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759)
>> >>
>> >> at
>> >>
>> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757)
>> >>
>> >> at
>> >>
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799)
>> >>
>> >> at
>> >>
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756)
>> >>
>> >> at
>> >>
>> org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)
>> >>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>
>>
>
> --
> With Regards,
> Andrew Carr
>
> e. andrewlanecarr@gmail.com
> w. andrew.carr@openlogic.com
> c. 4239489206
> a. P.O. Box 1231, Greeneville, TN, 37744
>


-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message