activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher L. Shannon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-5785) Deadlock between NIO worker and Broker.Servic threads
Date Wed, 20 May 2015 20:19:02 GMT

    [ https://issues.apache.org/jira/browse/AMQ-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553030#comment-14553030
] 

Christopher L. Shannon commented on AMQ-5785:
---------------------------------------------

This appears to probably be the same race condition issue I reported here: https://issues.apache.org/jira/browse/AMQ-5712

[~tabish121] uploaded a patch to that Jira that I've been running for a while and it has solved
the deadlock problem.  It hasn't been committed to master yet but if you don't mind applying
the patch yourself it should at least help you stop the deadlock until it gets fixed permanently.


> Deadlock between NIO worker and Broker.Servic threads
> -----------------------------------------------------
>
>                 Key: AMQ-5785
>                 URL: https://issues.apache.org/jira/browse/AMQ-5785
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.10.0
>         Environment: Physical Machine (192 GB RAM, 24 VCPU), RHEL 5.9, Java 1.7
> ActiveMQ runs on 4 GB heap
>            Reporter: Sree Panchajanyam D
>            Priority: Critical
>         Attachments: threaddump16214.txt
>
>
> During the peak loads we are encountering a recurring deadlock issue in ActiveMQ broker.
The threads that are deadlocked are 
> ActiveMQ  NIO Worker - trying to add message to FilePendingCursor
> Broker.Service Worker - that is trying to expire message from FilePendingCursor.
> =============================
> Found one Java-level deadlock:
> =============================
> "ActiveMQ NIO Worker 1003":
>   waiting to lock monitor 0x00002aeeb515a4f8 (object 0x00000007807da3e8, a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
>   which is held by "ActiveMQ BrokerService.worker.1"
> "ActiveMQ BrokerService.worker.1":
>   waiting for ownable synchronizer 0x000000077ac84b40, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "ActiveMQ NIO Worker 1003"
> Java stack information for the threads listed above:
> ===================================================
> "ActiveMQ NIO Worker 1003":
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
> 	- waiting to lock <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:96)
> 	- locked <0x00000007784e8c88> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
> 	at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1855)
> 	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:939)
> 	at org.apache.activemq.broker.region.Queue.send(Queue.java:733)
> 	at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:424)
> 	at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
> 	at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
> 	at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> 	at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
> 	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
> 	at org.apache.activemq.broker.UserIDBroker.send(UserIDBroker.java:56)
> 	at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:152)
> 	at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:496)
> 	at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:756)
> 	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> 	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
> 	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> 	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> 	at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> 	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> 	at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
> 	at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> 	at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
> 	at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> "ActiveMQ BrokerService.worker.1":
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000077ac84b40> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> 	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
> 	at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1841)
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.expireOldMessages(FilePendingMessageCursor.java:420)
> 	- locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:398)
> 	- locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at org.apache.activemq.usage.Usage$1.run(Usage.java:304)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message