activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mats Henrikson (JIRA)" <j...@apache.org>
Subject [jira] Created: (AMQ-2332) it's possible to cause a deadlock in ActiveMQ 5.2 using the client
Date Mon, 27 Jul 2009 22:29:33 GMT
it's possible to cause a deadlock in ActiveMQ 5.2 using the client
------------------------------------------------------------------

                 Key: AMQ-2332
                 URL: https://issues.apache.org/activemq/browse/AMQ-2332
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.2.0
            Reporter: Mats Henrikson


Hi,

We have found a deadlock in ActiveMQ 5.2. It's possible that we may be causing it by a multi-threading
problem in the client, we are currently looking into this. However, I feel that this is still
a bug in the broker since it shouldn't be possible to deadlock the broker by incorrect use
of the client.

The situation is that we have two clients connected to the broker, one over TCP using the
Java library, and one connecting over SSL (our own implementation, this may also be involved
in creating the problem) using the NMS library in .NET. 

The Java client connects fine and subscribes to some topics and queues and starts sending
messages to the topics. The NMS client connects and attempts to connect to some of the same
topics, as well as creating a temporary queue. As it is doing this initial setup stuff it
hangs and goes no further. From this point I don't think any more messages get delivered on
the topic, and Java clients trying to send messages to it hang on the producers send() call
indefinitely.

The deadlock appears to occur because a thread is stuck actually writing data back to the
NMS client (the bottom stack trace below), another thread (originating from the Java client)
is trying to acquire the write lock so that it too can write to the client (middle stack trace)
while still owning a lock on the list of consumers of a topic, meanwhile a third thread (this
one origination from the NMS client again) is trying to get that lock on the list of consumers
in order to add to it (top stack trace). For some reason the first thread trying to write
to the NMS client never finishes, even when the NMS client is killed, and so the lock is never
released and everything involving that topic grinds to a halt. 

I have looked but haven't been able to find anybody else reporting this issue.

I enabled JMX as well as debug mode and got the following stack traces from jconsole, and
tracked down the locks involved in the deadlock using a debugger:


Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
State: BLOCKED on java.util.concurrent.CopyOnWriteArrayList@509c6c30 owned by: ActiveMQ Transport:
tcp:///192.168.32.73:45666
Stack trace: 
org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
java.lang.Thread.run(Thread.java:595)


CopyOnWriteArrayList@509c6c30 lock acquired at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
below:

Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
State: BLOCKED on java.lang.Object@58c1c2f1 owned by: TempQueue:  c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
Stack trace: 
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
org.apache.activemq.broker.region.Topic.send(Topic.java:365)
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
java.lang.Thread.run(Thread.java:595)


Object@58c1c2f1 lock acquired at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
below:

Name: TempQueue:  c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
State: RUNNABLE
Total blocked: 55  Total waited: 11
Stack trace: 
java.net.SocketOutputStream.socketWrite0(Native Method)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
java.io.DataOutputStream.flush(DataOutputStream.java:106)
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

Let me know if I can do anything else to help.

--
Mats Henrikson
Software Engineer
SunGard Kiodex


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message