activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raul Kripalani <r...@evosent.com>
Subject Re: Deadlock on activemq startup….
Date Fri, 18 Jul 2014 13:41:48 GMT
Hi,

We found a similar issue some time ago and it was due to a nasty
interaction between optimized dispatch and the JMS scheduler.

Is optimizedDispatch = true by any chance in the affected destination?

Thanks,

*Raúl Kripalani*
Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
Integration specialist
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

On Fri, Jul 18, 2014 at 3:33 AM, Kevin Burton <burton@spinn3r.com> wrote:

> This is mildly unfortunately.  Probably should migrate to LevelDB… :)
>
> But this is definitely a bug.
>
> Essentially what happened was that I tried to purge a queue with a large
> number of message.  ActiveMQ went to 100% CPU and stopped processing
> messages or making any progress.
>
> I stopped ActiveMQ and then restarted it… and then it deadlocked.
>
> Found one Java-level deadlock:
> =============================
> "JobScheduler:JMS":
>   waiting for ownable synchronizer 0x00000000f6c60fa8, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "main"
> "main":
>   waiting for ownable synchronizer 0x00000000c0027260, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "JobScheduler:JMS"
>
> Java stack information for the threads listed above:
> ===================================================
> "JobScheduler:JMS":
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000f6c60fa8> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>  at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
>  at
>
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
> at
>
> org.apache.activemq.broker.region.AbstractRegion.getDestinations(AbstractRegion.java:243)
>  at
>
> org.apache.activemq.broker.region.RegionBroker.getDestinations(RegionBroker.java:140)
> at
>
> org.apache.activemq.store.kahadb.KahaDBStore.rollbackStatsOnDuplicate(KahaDBStore.java:284)
>  at
>
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1316)
> at
>
> org.apache.activemq.store.kahadb.MessageDatabase$11.execute(MessageDatabase.java:1140)
>  at
>
> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
> at
>
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1137)
>  at
>
> org.apache.activemq.store.kahadb.MessageDatabase$10.visit(MessageDatabase.java:1074)
> at
>
> org.apache.activemq.store.kahadb.data.KahaAddMessageCommand.visit(KahaAddMessageCommand.java:241)
>  at
>
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1071)
> at
>
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:978)
>  at
>
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:958)
> at
>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:424)
>  at
>
> org.apache.activemq.store.kahadb.KahaDBTransactionStore.addMessage(KahaDBTransactionStore.java:388)
> at
>
> org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.addMessage(KahaDBTransactionStore.java:160)
>  at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:913)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:733)
>  at
>
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:424)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
>  at
>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
>  at
>
> org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:282)
> at
>
> org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:485)
>  at
>
> org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:535)
> at
>
> org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:492)
>  at java.lang.Thread.run(Thread.java:724)
> "main":
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x00000000c0027260> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>  at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
>  at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at
>
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
>  at
>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.resetBatching(KahaDBStore.java:618)
> at
>
> org.apache.activemq.store.ProxyMessageStore.resetBatching(ProxyMessageStore.java:111)
>  at
>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.resetBatch(QueueStorePrefetch.java:92)
> at
>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.start(AbstractStoreCursor.java:61)
>  - locked <0x00000000f98a12a8> (a
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch)
> at
>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.start(StoreQueueCursor.java:72)
>  - locked <0x00000000f98a1258> (a
> org.apache.activemq.broker.region.cursors.StoreQueueCursor)
> at org.apache.activemq.broker.region.Queue.start(Queue.java:1037)
>  at
>
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:141)
> at
>
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:325)
>  at
>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:167)
> at
>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:167)
>  at
>
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:184)
> at
>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:167)
>  at
>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:167)
> at
>
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:172)
>  at
>
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> at
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:190)
>  at
>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:182)
>  at
>
> org.apache.activemq.broker.scheduler.SchedulerBroker.start(SchedulerBroker.java:81)
> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:182)
>  at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:182)
> at
>
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:120)
>  at
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2177)
> at
>
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:675)
>  at
>
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:659)
> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:595)
>  at
>
> org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1638)
>  at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1579)
> at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)
>  at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
> at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
>  at
>
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)
> at
>
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
>  - locked <0x00000000c0051148> (a java.util.concurrent.ConcurrentHashMap)
> at
>
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
>  at
>
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> at
>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)
>  at
>
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
> at
>
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
>  - locked <0x00000000c0051240> (a java.lang.Object)
> at
>
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
>  at
>
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
> at
>
> org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)
>  at
>
> org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)
> at
>
> org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)
>  at
>
> org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
> at
>
> org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
>  at
>
> org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)
> at
>
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
>  at
>
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
> at
>
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
>  at
>
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
>  at org.apache.activemq.console.Main.main(Main.java:115)
>
> Found 1 deadlock.
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message