Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: softfail (nike.apache.org: transitioning domain of
 claudio.santana@gmail.com does not designate 162.253.133.43 as permitted
 sender)
Date: Thu, 7 Aug 2014 17:53:55 -0700 (PDT)
From: pollotek <claudio.santana@gmail.com>
To: users@activemq.apache.org
Message-ID: <1407459235343-4684288.post@n4.nabble.com>
Subject: Deadlock with ActiveMQ 5.6
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hi, I'm been having random problems with my ActiveMQ for a while now. I have
this architecture:

Producers with embedded broker ---> network of non-persistent brokers --->
consumers

I'm using ActiveMQ 5.6 with KahaDB. The producers and consumers run inside
Tomcat 6 with Java 7. I poll stats from these servers with jmxtrans every 5
minutes.

I have been experiencing issues randomly for a while where the stand alone
brokers stop dispatching messages to the consumers for specific queues which
causes storage to fill up on the stand alone broker causing all producers
with embedded brokers to buffer messages. Eventually the embedded broker
storage also fills up and causes lots of trouble.

I took a thread dump on two of the servers that were giving me a hard time
and one stand alone brokers (the other one for some reason jstack couldn't
attach to the process).

In one of the embedded brokers I found this deadlock detected with TDA:

"ActiveMQ Task-100" daemon prio=10 tid=0x00007f01e8050000 nid=0x19d8 waiting
on condition [0x00007efeb1a7f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000069d260d18> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
	at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
	at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
	at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:144)
	at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
	at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135)
	at org.apache.activemq.transport.vm.VMTransport.start(VMTransport.java:156)
	- locked <0x000000069d265f50> (a java.util.concurrent.atomic.AtomicBoolean)
	at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
	at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
	at
org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:914)
	- locked <0x000000069d266018> (a
org.apache.activemq.broker.jmx.ManagedTransportConnection)
	at
org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:227)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

"ActiveMQ Task-81" daemon prio=10 tid=0x00007f016c046800 nid=0x19c9 waiting
for monitor entry [0x00007efeb278c000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at
org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1022)
	- waiting to lock <0x000000069d266018> (a
org.apache.activemq.broker.jmx.ManagedTransportConnection)
	at
org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:996)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
	
A few questions:
- Is this a known issue (I couldn't find anything in previous posts)? Is
there a fix for it in a newer version? Maybe there's a workaround that
doesn't require updating to a higher version number and still use jmx?
- Would this deadlock cause messages to not be forwarded all the way to the
stand alone brokers and consumers? I can share the full config of the
individual brokers if that would help.

In the worst cases all this spirals out of control and I end up having to
clear the KahaDB storage of all my embedded brokers. Terrible situation.

I really hope somebody can help me out.

Thanks,
Claudio


--
View this message in context: http://activemq.2283324.n4.nabble.com/Deadlock-with-ActiveMQ-5-6-tp4684288.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.