activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From drvillo <f.viv...@gmail.com>
Subject Random deadlocks or lost messages with AMQ 4.1 and Jencks
Date Fri, 15 Dec 2006 16:41:11 GMT

Hi there

I have spent the whole day now tracking down this, and I have no clue.
I have stripped everything down to a silly system:
sender1->queue1->listener1->sender2->queue2->listener2


I have a test that basically sends 10 messages with sender1, no more.

The listeners are just logging the incoming messages, and listener1 has 
a sender himself, to resend the same message to the second queue.


Actually the situation is the following
-sometimes I don't manage to send all the messages, because of a deadlock
(given below)
-sometimes I get no deadlocks but I loose a message (always the second) (and
from
the jconsole I can see that there's a message stuck in the queue and not
consumed)
-sometimes it goes fine


the deadlock is reported below, with the last logging information appeared

DEBUG org.apache.activemq.broker.region.AbstractRegion - Removing consumer:
ID:obimac.local-51149-1166199529487-2:2:-1:4
DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - run loop end
DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - Session returned to
pool: ServerSessionImpl:1
DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Run finished
DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Work completed:
javax.resource.spi.work.WorkEvent[source=Work :ServerSessionImpl:1]
DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - ServerSession
requested.


...

Found one Java-level deadlock:
=============================
"ActiveMQ Task":
  waiting to lock monitor 0x018123fc (object 0x26b8fa00, a
java.lang.Object),
  which is held by "Thread-3"
"Thread-3":
  waiting to lock monitor 0x018123d8 (object 0x26b8e848, a
java.lang.Object),
  which is held by "ActiveMQ Task"

Java stack information for the threads listed above:
===================================================
"ActiveMQ Task":
	at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
	- waiting to lock <0x26b8fa00> (a java.lang.Object)
	at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1151)
	at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:226)
	at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:277)
	at
org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl(ServerSessionPoolImpl.java:63)
	at
org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:116)
	at
org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:136)
	at
org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1473)
	at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
	at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
	at
org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
	at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
	at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
	- locked <0x26b8e848> (a java.lang.Object)
	at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
	at
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:772)
	at
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:788)
	at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:111)
	at
org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
	at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44)
	at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
	at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
	at java.lang.Thread.run(Thread.java:613)
"Thread-3":
	at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
	- waiting to lock <0x26b8e848> (a java.lang.Object)
	at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
	at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
	at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
	at
org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
	at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
	at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
	- locked <0x26b8fa00> (a java.lang.Object)
	at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
	at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74)
	at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1171)
	at
org.apache.activemq.TransactionContext.commit(TransactionContext.java:260)
	at
org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:58)
	at
org.jencks.LocalTransactionEndpoint.afterDelivery(LocalTransactionEndpoint.java:52)
	at
org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
	at
org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
	at
org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:216)
	at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
	at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
	at
org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
	at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
	at java.lang.Thread.run(Thread.java:613)

Found 1 deadlock.


Attached there's my spring config file. I have been trying switching 
async dispatches and sends off, thinking that could have been the problem
but that
didn't help. turning off message persistence doesn't change anything either.


Needless to say that the whole of this is pretty frustrating, in the last
couple of days I have been
dealing with deadlocks all over the place.
I'm running AMC 4.1, jencks 1.3 spring 2.0.

I hope someone will help on this, otherwise I'll probably have to drop the
whole project:(

thanks
Francesco Vivoli

http://www.nabble.com/file/4735/container.xml container.xml 
-- 
View this message in context: http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7894874
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message