activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Timothy Bish (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (AMQ-5219) Deadlock between JMS Job Scheduler and broker initialization
Date Mon, 23 Jun 2014 16:01:25 GMT

     [ https://issues.apache.org/jira/browse/AMQ-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Timothy Bish closed AMQ-5219.
-----------------------------

    Resolution: Incomplete

This is reported against an old release with no tests to validate.  Fixes to scheduler startup
have gone into the later releases.  Please reopen if you can produce a test that shows there
is still an issue.  

> Deadlock between JMS Job Scheduler and broker initialization
> ------------------------------------------------------------
>
>                 Key: AMQ-5219
>                 URL: https://issues.apache.org/jira/browse/AMQ-5219
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.7.0
>         Environment: Apache SMX 4.5.0, JDK 1.6, RHEL.
>            Reporter: Raúl Kripalani
>
> Thread dump exhibiting a deadlock shows that JMS scheduler is started too early in the
broker boot procedure. If scheduled jobs are pending and fired straightaway after scheduler
initialisation – while other broker elements are still being initialised – this can incur
in a deadlock which stalls the broker completely.
> {code}
> Java stack information for the threads listed above:
> ===================================================
> "JobScheduler:JMS":
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000006bca4ecb0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
> 	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
> 	at org.apache.activemq.broker.region.AbstractRegion.getDestinations(AbstractRegion.java:240)
> 	at org.apache.activemq.broker.region.RegionBroker.getDestinations(RegionBroker.java:130)
> 	at org.apache.activemq.store.kahadb.KahaDBStore.rollbackStatsOnDuplicate(KahaDBStore.java:295)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.upadateIndex(MessageDatabase.java:1165)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$14.execute(MessageDatabase.java:982)
> 	at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:980)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:927)
> 	at org.apache.activemq.store.kahadb.data.KahaAddMessageCommand.visit(KahaAddMessageCommand.java:241)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:924)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:836)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:818)
> 	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:433)
> 	at org.apache.activemq.store.kahadb.KahaDBTransactionStore.addMessage(KahaDBTransactionStore.java:385)
> 	at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.addMessage(KahaDBTransactionStore.java:157)
> 	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:746)
> 	at org.apache.activemq.broker.region.Queue.send(Queue.java:717)
> 	at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
> 	at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
> 	at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:311)
> 	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> 	at org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:251)
> 	at org.apache.activemq.broker.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:423)
> 	at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:473)
> 	- locked <0x00000006bca05f50> (a org.apache.activemq.broker.scheduler.JobSchedulerImpl)
> 	at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> 	at java.lang.Thread.run(Thread.java:662)
> "Start Level Event Dispatcher":
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000006bc5cd318> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
> 	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
> 	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.getMessageCount(KahaDBStore.java:477)
> 	at org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:101)
> 	at org.apache.activemq.broker.region.Queue.initialize(Queue.java:376)
> 	at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:86)
> 	at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> 	at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> 	at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> 	at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> 	- locked <0x00000006bca82920> (a java.util.concurrent.ConcurrentHashMap)
> 	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> 	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> 	at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> 	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> 	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> 	at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> 	at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> 	at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> 	at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> 	at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> 	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> 	at org.apache.activemq.broker.scheduler.SchedulerBroker.start(SchedulerBroker.java:89)
> 	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> 	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> 	at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> 	at org.apache.activemq.broker.MutableBrokerFilter.start(MutableBrokerFilter.java:163)
> 	at org.apache.activemq.broker.BrokerPluginSupport.start(BrokerPluginSupport.java:40)
> 	at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> 	at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> 	at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> 	at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> 	at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:58)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1546)
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1487)
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1419)
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:518)
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:455)
> 	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
> 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
> 	- locked <0x00000006bc3cb720> (a java.util.concurrent.ConcurrentHashMap)
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
> 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
> 	- locked <0x00000006bc3cb510> (a java.util.concurrent.ConcurrentHashMap)
> 	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
> 	at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:69)
> 	at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:355)
> 	- locked <0x00000006bc3bccc0> (a java.lang.Object)
> 	at org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
> 	at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:320)
> 	at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:132)
> 	at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
> 	at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageTwo(DependencyWaiterApplicationContextExecutor.java:299)
> 	at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:244)
> 	at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:169)
> 	at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:175)
> 	at org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:716)
> 	at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
> 	at org.springframework.osgi.extender.internal.activator.ContextLoaderListener.maybeCreateApplicationContextFor(ContextLoaderListener.java:781)
> 	at org.springframework.osgi.extender.internal.activator.ContextLoaderListener$ContextBundleListener.handleEvent(ContextLoaderListener.java:229)
> 	at org.springframework.osgi.extender.internal.activator.ContextLoaderListener$BaseListener.bundleChanged(ContextLoaderListener.java:172)
> 	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:919)
> 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:227)
> 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:149)
> 	at org.eclipse.osgi.framework.internal.core.Framework.publishBundleEventPrivileged(Framework.java:1349)
> 	at org.eclipse.osgi.framework.internal.core.Framework.publishBundleEvent(Framework.java:1300)
> 	at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:380)
> 	at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:374)
> 	at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1067)
> 	at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:561)
> 	at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:546)
> 	at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:459)
> 	at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243)
> 	- locked <0x00000006ba9f9338> (a java.lang.Object)
> 	at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:440)
> 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:227)
> 	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:337)
> Found 1 deadlock.
> {code}
> Useful info:
> * This was seen in an OSGi environment, Apache SMX 4.5.0.
> * Broker is being initialised through Spring configuration. Original OOTB Blueprint activemq-broker.xml
was replaced with Spring equivalent, with the following manifest option:
> {code}
>     <manifest>
>         Spring-Context: *;publish-context:=false;create-asynchronously:=false
>     </manifest>
> {code}
> ** This ensures that the broker is started before all other business bundles (by starting
this Spring context synchronously, the OSGi framework waits until the bundle is started before
continuing with starting the rest).
> * No other broker plugins apart from the redeliveryPlugin are configured. Config of this
one is as follows:
> {code}
> <redeliveryPlugin fallbackToDeadLetter="true" sendToDlqIfMaxRetriesExceeded="true">
>                 <redeliveryPolicyMap>
>                     <redeliveryPolicyMap>
>                         <redeliveryPolicyEntries>
> ...
>                         </redeliveryPolicyEntries>
>                     </redeliveryPolicyMap>
>                 </redeliveryPolicyMap>
>             </redeliveryPlugin>
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message