activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From drvillo <f.viv...@gmail.com>
Subject Re: Random deadlocks or lost messages with AMQ 4.1 and Jencks
Date Mon, 18 Dec 2006 10:17:43 GMT

Hi 

actually running with async turned on,
<property name="serverUrl"
value="vm://brokerConfigTest?jms.useAsyncSend=true&amp;jms.dispatchAsync=true"/>

allows the test to run for 10 messages, but results in another deadlock to
be found
when trying with 1000 messages...

the above URI should be fine right? it is given to the resource adapter.


Thanks again, I'm trying now with tcp instead of the vm transport...


Found one Java-level deadlock:
=============================
"Thread-6":
  waiting to lock monitor 0x018123d8 (object 0x26b8f0a0, a
java.lang.Object),
  which is held by "ActiveMQ Task"
"ActiveMQ Task":
  waiting to lock monitor 0x018123b4 (object 0x26b8e7b0, a
java.lang.Object),
  which is held by "Thread-6"

Java stack information for the threads listed above:
===================================================
"Thread-6":
	at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:42)
	- waiting to lock <0x26b8f0a0> (a java.lang.Object)
	at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
	at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
	at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
	at
org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
	at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
	at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
	- locked <0x26b8e7b0> (a java.lang.Object)
	at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
	at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74)
	at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1171)
	at
org.apache.activemq.TransactionContext.commit(TransactionContext.java:260)
	at
org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:58)
	at
org.jencks.LocalTransactionEndpoint.afterDelivery(LocalTransactionEndpoint.java:52)
	at
org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
	at
org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
	at
org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:216)
	at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
	at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
	at
org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
	at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
	at java.lang.Thread.run(Thread.java:613)
"ActiveMQ Task":
	at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:42)
	- waiting to lock <0x26b8e7b0> (a java.lang.Object)
	at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1151)
	at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:226)
	at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:277)
	at
org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl(ServerSessionPoolImpl.java:63)
	at
org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:116)
	at
org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:136)
	at
org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1473)
	at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
	at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
	at
org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
	at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
	at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
	- locked <0x26b8f0a0> (a java.lang.Object)
	at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
	at
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:772)
	at
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:788)
	at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:111)
	at
org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
	at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44)
	at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
	at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
	at java.lang.Thread.run(Thread.java:613)

Found 1 deadlock.



rajdavies wrote:
> 
> As you're using the vm://transport - you could set the async flag =  
> true - which will cure this problem - e.g. your connection url would  
> be vm://localhost?async=true
> 
> cheers,
> 
> Rob
> On 15 Dec 2006, at 16:41, drvillo wrote:
> 
>>
>> Hi there
>>
>> I have spent the whole day now tracking down this, and I have no clue.
>> I have stripped everything down to a silly system:
>> sender1->queue1->listener1->sender2->queue2->listener2
>>
>>
>> I have a test that basically sends 10 messages with sender1, no more.
>>
>> The listeners are just logging the incoming messages, and listener1  
>> has
>> a sender himself, to resend the same message to the second queue.
>>
>>
>> Actually the situation is the following
>> -sometimes I don't manage to send all the messages, because of a  
>> deadlock
>> (given below)
>> -sometimes I get no deadlocks but I loose a message (always the  
>> second) (and
>> from
>> the jconsole I can see that there's a message stuck in the queue  
>> and not
>> consumed)
>> -sometimes it goes fine
>>
>>
>> the deadlock is reported below, with the last logging information  
>> appeared
>>
>> DEBUG org.apache.activemq.broker.region.AbstractRegion - Removing  
>> consumer:
>> ID:obimac.local-51149-1166199529487-2:2:-1:4
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - run loop end
>> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - Session  
>> returned to
>> pool: ServerSessionImpl:1
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Run finished
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Work completed:
>> javax.resource.spi.work.WorkEvent[source=Work :ServerSessionImpl:1]
>> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - ServerSession
>> requested.
>>
>>
>> ...
>>
>> Found one Java-level deadlock:
>> =============================
>> "ActiveMQ Task":
>>   waiting to lock monitor 0x018123fc (object 0x26b8fa00, a
>> java.lang.Object),
>>   which is held by "Thread-3"
>> "Thread-3":
>>   waiting to lock monitor 0x018123d8 (object 0x26b8e848, a
>> java.lang.Object),
>>   which is held by "ActiveMQ Task"
>>
>> Java stack information for the threads listed above:
>> ===================================================
>> "ActiveMQ Task":
>> 	at
>> org.apache.activemq.transport.MutexTransport.oneway 
>> (MutexTransport.java:43)
>> 	- waiting to lock <0x26b8fa00> (a java.lang.Object)
>> 	at
>> org.apache.activemq.transport.ResponseCorrelator.oneway 
>> (ResponseCorrelator.java:60)
>> 	at
>> org.apache.activemq.ActiveMQConnection.asyncSendPacket 
>> (ActiveMQConnection.java:1151)
>> 	at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java: 
>> 226)
>> 	at
>> org.apache.activemq.ActiveMQConnection.createSession 
>> (ActiveMQConnection.java:277)
>> 	at
>> org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl 
>> (ServerSessionPoolImpl.java:63)
>> 	at
>> org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession 
>> (ServerSessionPoolImpl.java:116)
>> 	at
>> org.apache.activemq.ActiveMQConnectionConsumer.dispatch 
>> (ActiveMQConnectionConsumer.java:136)
>> 	at
>> org.apache.activemq.ActiveMQConnection.onCommand 
>> (ActiveMQConnection.java:1473)
>> 	at
>> org.apache.activemq.transport.ResponseCorrelator.onCommand 
>> (ResponseCorrelator.java:95)
>> 	at
>> org.apache.activemq.transport.TransportFilter.onCommand 
>> (TransportFilter.java:65)
>> 	at
>> org.apache.activemq.transport.vm.VMTransport.syncOneWay 
>> (VMTransport.java:96)
>> 	at org.apache.activemq.transport.vm.VMTransport.oneway 
>> (VMTransport.java:83)
>> 	at
>> org.apache.activemq.transport.MutexTransport.oneway 
>> (MutexTransport.java:43)
>> 	- locked <0x26b8e848> (a java.lang.Object)
>> 	at
>> org.apache.activemq.transport.ResponseCorrelator.oneway 
>> (ResponseCorrelator.java:60)
>> 	at
>> org.apache.activemq.broker.TransportConnection.dispatch 
>> (TransportConnection.java:1051)
>> 	at
>> org.apache.activemq.broker.TransportConnection.processDispatch 
>> (TransportConnection.java:772)
>> 	at
>> org.apache.activemq.broker.TransportConnection.iterate 
>> (TransportConnection.java:788)
>> 	at
>> org.apache.activemq.thread.PooledTaskRunner.runTask 
>> (PooledTaskRunner.java:111)
>> 	at
>> org.apache.activemq.thread.PooledTaskRunner.access$100 
>> (PooledTaskRunner.java:26)
>> 	at
>> org.apache.activemq.thread.PooledTaskRunner$1.run 
>> (PooledTaskRunner.java:44)
>> 	at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor 
>> $Worker.runTask(ThreadPoolExecutor.java:650)
>> 	at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor 
>> $Worker.run(ThreadPoolExecutor.java:675)
>> 	at java.lang.Thread.run(Thread.java:613)
>> "Thread-3":
>> 	at
>> org.apache.activemq.transport.MutexTransport.oneway 
>> (MutexTransport.java:43)
>> 	- waiting to lock <0x26b8e848> (a java.lang.Object)
>> 	at
>> org.apache.activemq.transport.ResponseCorrelator.oneway 
>> (ResponseCorrelator.java:60)
>> 	at
>> org.apache.activemq.broker.TransportConnection.dispatch 
>> (TransportConnection.java:1051)
>> 	at
>> org.apache.activemq.broker.TransportConnection$1.onCommand 
>> (TransportConnection.java:179)
>> 	at
>> org.apache.activemq.transport.ResponseCorrelator.onCommand 
>> (ResponseCorrelator.java:95)
>> 	at
>> org.apache.activemq.transport.TransportFilter.onCommand 
>> (TransportFilter.java:65)
>> 	at
>> org.apache.activemq.transport.vm.VMTransport.syncOneWay 
>> (VMTransport.java:96)
>> 	at org.apache.activemq.transport.vm.VMTransport.oneway 
>> (VMTransport.java:83)
>> 	at
>> org.apache.activemq.transport.MutexTransport.oneway 
>> (MutexTransport.java:43)
>> 	- locked <0x26b8fa00> (a java.lang.Object)
>> 	at
>> org.apache.activemq.transport.ResponseCorrelator.asyncRequest 
>> (ResponseCorrelator.java:69)
>> 	at
>> org.apache.activemq.transport.ResponseCorrelator.request 
>> (ResponseCorrelator.java:74)
>> 	at
>> org.apache.activemq.ActiveMQConnection.syncSendPacket 
>> (ActiveMQConnection.java:1171)
>> 	at
>> org.apache.activemq.TransactionContext.commit 
>> (TransactionContext.java:260)
>> 	at
>> org.apache.activemq.ra.LocalAndXATransaction.commit 
>> (LocalAndXATransaction.java:58)
>> 	at
>> org.jencks.LocalTransactionEndpoint.afterDelivery 
>> (LocalTransactionEndpoint.java:52)
>> 	at
>> org.apache.activemq.ra.MessageEndpointProxy 
>> $MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
>> 	at
>> org.apache.activemq.ra.MessageEndpointProxy.afterDelivery 
>> (MessageEndpointProxy.java:65)
>> 	at
>> org.apache.activemq.ra.ServerSessionImpl.afterDelivery 
>> (ServerSessionImpl.java:216)
>> 	at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
>> 	at org.apache.activemq.ra.ServerSessionImpl.run 
>> (ServerSessionImpl.java:165)
>> 	at
>> org.apache.geronimo.connector.work.WorkerContext.run 
>> (WorkerContext.java:291)
>> 	at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
>> Source)
>> 	at java.lang.Thread.run(Thread.java:613)
>>
>> Found 1 deadlock.
>>
>>
>> Attached there's my spring config file. I have been trying switching
>> async dispatches and sends off, thinking that could have been the  
>> problem
>> but that
>> didn't help. turning off message persistence doesn't change  
>> anything either.
>>
>>
>> Needless to say that the whole of this is pretty frustrating, in  
>> the last
>> couple of days I have been
>> dealing with deadlocks all over the place.
>> I'm running AMC 4.1, jencks 1.3 spring 2.0.
>>
>> I hope someone will help on this, otherwise I'll probably have to  
>> drop the
>> whole project:(
>>
>> thanks
>> Francesco Vivoli
>>
>> http://www.nabble.com/file/4735/container.xml container.xml
>> -- 
>> View this message in context: http://www.nabble.com/Random- 
>> deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks- 
>> tf2828116.html#a7894874
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7926348
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message