activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sree Panchajanyam D (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMQ-5785) Deadlock between NIO worker and Broker.Servic threads
Date Wed, 20 May 2015 13:25:00 GMT

     [ https://issues.apache.org/jira/browse/AMQ-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sree Panchajanyam D updated AMQ-5785:
-------------------------------------
    Attachment: threaddump16214.txt

Attaching the thread dump. This issue is occurring everyday at peak hours. We have upgraded
to 5.10.0 from 5.6.0 about 4 months ago. This issue has started occurring since a week.  

> Deadlock between NIO worker and Broker.Servic threads
> -----------------------------------------------------
>
>                 Key: AMQ-5785
>                 URL: https://issues.apache.org/jira/browse/AMQ-5785
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.10.0
>         Environment: Physical Machine (192 GB RAM, 24 VCPU), RHEL 5.9, Java 1.7
> ActiveMQ runs on 4 GB heap
>            Reporter: Sree Panchajanyam D
>            Priority: Critical
>         Attachments: threaddump16214.txt
>
>
> During the peak loads we are encountering a recurring deadlock issue in ActiveMQ broker.
The threads that are deadlocked are 
> ActiveMQ  NIO Worker - trying to add message to FilePendingCursor
> Broker.Service Worker - that is trying to expire message from FilePendingCursor.
> =============================
> Found one Java-level deadlock:
> =============================
> "ActiveMQ NIO Worker 1003":
>   waiting to lock monitor 0x00002aeeb515a4f8 (object 0x00000007807da3e8, a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
>   which is held by "ActiveMQ BrokerService.worker.1"
> "ActiveMQ BrokerService.worker.1":
>   waiting for ownable synchronizer 0x000000077ac84b40, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "ActiveMQ NIO Worker 1003"
> Java stack information for the threads listed above:
> ===================================================
> "ActiveMQ NIO Worker 1003":
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
> 	- waiting to lock <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:96)
> 	- locked <0x00000007784e8c88> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
> 	at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1855)
> 	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:939)
> 	at org.apache.activemq.broker.region.Queue.send(Queue.java:733)
> 	at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:424)
> 	at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
> 	at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
> 	at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> 	at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
> 	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
> 	at org.apache.activemq.broker.UserIDBroker.send(UserIDBroker.java:56)
> 	at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:152)
> 	at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:496)
> 	at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:756)
> 	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> 	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
> 	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> 	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> 	at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> 	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> 	at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
> 	at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> 	at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
> 	at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> "ActiveMQ BrokerService.worker.1":
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000077ac84b40> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> 	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
> 	at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1841)
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.expireOldMessages(FilePendingMessageCursor.java:420)
> 	- locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:398)
> 	- locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at org.apache.activemq.usage.Usage$1.run(Usage.java:304)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message