activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sree Panchajanyam D (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-5785) Deadlock between NIO worker and Broker.Servic threads
Date Tue, 26 May 2015 07:23:17 GMT

    [ https://issues.apache.org/jira/browse/AMQ-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558776#comment-14558776
] 

Sree Panchajanyam D commented on AMQ-5785:
------------------------------------------

Agree with Timothy. This seems to be a different issue. This issue occurs in one of our several
AMQ environments and cannot be replicated anywhere else.

I can go with the upgrade if I can find a root cause of the issue and for sure know that 5.11.1
will fix it as an AMQ upgrade in our environment is non trivial.
The issue again occurred yesterday and the thread dumps from yesterday tell the same story.
If you need any activemq logs either in trace/debug mode I will be able to attach them here.

I am going through the ActiveMQ source code to corner this issue. Any pointers/help in this
regard is greatly appreciated.

> Deadlock between NIO worker and Broker.Servic threads
> -----------------------------------------------------
>
>                 Key: AMQ-5785
>                 URL: https://issues.apache.org/jira/browse/AMQ-5785
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.10.0
>         Environment: Physical Machine (192 GB RAM, 24 VCPU), RHEL 5.9, Java 1.7
> ActiveMQ runs on 4 GB heap
>            Reporter: Sree Panchajanyam D
>            Priority: Critical
>         Attachments: threaddump16214.txt
>
>
> During the peak loads we are encountering a recurring deadlock issue in ActiveMQ broker.
The threads that are deadlocked are 
> ActiveMQ  NIO Worker - trying to add message to FilePendingCursor
> Broker.Service Worker - that is trying to expire message from FilePendingCursor.
> =============================
> Found one Java-level deadlock:
> =============================
> "ActiveMQ NIO Worker 1003":
>   waiting to lock monitor 0x00002aeeb515a4f8 (object 0x00000007807da3e8, a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
>   which is held by "ActiveMQ BrokerService.worker.1"
> "ActiveMQ BrokerService.worker.1":
>   waiting for ownable synchronizer 0x000000077ac84b40, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "ActiveMQ NIO Worker 1003"
> Java stack information for the threads listed above:
> ===================================================
> "ActiveMQ NIO Worker 1003":
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
> 	- waiting to lock <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:96)
> 	- locked <0x00000007784e8c88> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
> 	at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1855)
> 	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:939)
> 	at org.apache.activemq.broker.region.Queue.send(Queue.java:733)
> 	at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:424)
> 	at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
> 	at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
> 	at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> 	at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
> 	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
> 	at org.apache.activemq.broker.UserIDBroker.send(UserIDBroker.java:56)
> 	at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:152)
> 	at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:496)
> 	at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:756)
> 	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> 	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
> 	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> 	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> 	at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> 	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> 	at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
> 	at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> 	at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
> 	at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> "ActiveMQ BrokerService.worker.1":
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000077ac84b40> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> 	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
> 	at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1841)
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.expireOldMessages(FilePendingMessageCursor.java:420)
> 	- locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:398)
> 	- locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at org.apache.activemq.usage.Usage$1.run(Usage.java:304)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message