activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mats Henrikson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-2332) it's possible to cause a deadlock in ActiveMQ 5.2 using the client
Date Mon, 27 Jul 2009 22:43:33 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=52981#action_52981
] 

Mats Henrikson commented on AMQ-2332:
-------------------------------------

Versions for the broker:

ActiveMQ 5.2 release

java version "1.5.0_14"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)

The Java client is using the jar out of the broker package.

The broker and Java client are running on some RedHat Linux 64 bit.

The NMS client is version 1.2 I believe, and runs on Windows XP.

> it's possible to cause a deadlock in ActiveMQ 5.2 using the client
> ------------------------------------------------------------------
>
>                 Key: AMQ-2332
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2332
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.2.0
>            Reporter: Mats Henrikson
>
> Hi,
> We have found a deadlock in ActiveMQ 5.2. It's possible that we may be causing it by
a multi-threading problem in the client, we are currently looking into this. However, I feel
that this is still a bug in the broker since it shouldn't be possible to deadlock the broker
by incorrect use of the client.
> The situation is that we have two clients connected to the broker, one over TCP using
the Java library, and one connecting over SSL (our own implementation, this may also be involved
in creating the problem) using the NMS library in .NET. 
> The Java client connects fine and subscribes to some topics and queues and starts sending
messages to the topics. The NMS client connects and attempts to connect to some of the same
topics, as well as creating a temporary queue. As it is doing this initial setup stuff it
hangs and goes no further. From this point I don't think any more messages get delivered on
the topic, and Java clients trying to send messages to it hang on the producers send() call
indefinitely.
> The deadlock appears to occur because a thread is stuck actually writing data back to
the NMS client (the bottom stack trace below), another thread (originating from the Java client)
is trying to acquire the write lock so that it too can write to the client (middle stack trace)
while still owning a lock on the list of consumers of a topic, meanwhile a third thread (this
one origination from the NMS client again) is trying to get that lock on the list of consumers
in order to add to it (top stack trace). For some reason the first thread trying to write
to the NMS client never finishes, even when the NMS client is killed, and so the lock is never
released and everything involving that topic grinds to a halt. 
> I have looked but haven't been able to find anybody else reporting this issue.
> I enabled JMX as well as debug mode and got the following stack traces from jconsole,
and tracked down the locks involved in the deadlock using a debugger:
> Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
> State: BLOCKED on java.util.concurrent.CopyOnWriteArrayList@509c6c30 owned by: ActiveMQ
Transport: tcp:///192.168.32.73:45666
> Stack trace: 
> org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
> org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> CopyOnWriteArrayList@509c6c30 lock acquired at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
below:
> Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
> State: BLOCKED on java.lang.Object@58c1c2f1 owned by: TempQueue:  c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> Stack trace: 
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
> org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
> org.apache.activemq.broker.region.Topic.send(Topic.java:365)
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> Object@58c1c2f1 lock acquired at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
below:
> Name: TempQueue:  c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> State: RUNNABLE
> Total blocked: 55  Total waited: 11
> Stack trace: 
> java.net.SocketOutputStream.socketWrite0(Native Method)
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
> com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
> com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> java.io.DataOutputStream.flush(DataOutputStream.java:106)
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
> org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
> org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
> org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Let me know if I can do anything else to help.
> --
> Mats Henrikson
> Software Engineer
> SunGard Kiodex

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message