activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "matteo rulli (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AMQ-5260) Cross talk over duplex network connection can lead to blocking (alternative take)
Date Mon, 15 Sep 2014 10:06:35 GMT

    [ https://issues.apache.org/jira/browse/AMQ-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133752#comment-14133752
] 

matteo rulli edited comment on AMQ-5260 at 9/15/14 10:06 AM:
-------------------------------------------------------------

Ok, we managed to systematically reproduce the issue in a dedicated test-case (please find
it as an attachment to this issue, *AMQ5260AdvancedTest.java*). This junit must be inserted
in the _org.apache.activemq.bugs_ package within the activemq official src distro (version
5.10.0). In order to reproduce the deadlock in few seconds, the following configuration must
be applied (VMArgs for the runnable jar of the junit test launch config):
{noformat}
	-Dorg.apache.activemq.UseDedicatedTaskRunner=false 
	-Dbroker.ip.address=<broker ip goes here> 
	-Dnum.producers.spec.topic=5 
	-Dnum.consumers.spec.topic=5 
	-Dnum.producers.gen.topic=5 
	-Dnum.consumers.gen.topic=5 
	-Dnum.messages.sec=200
{noformat}

At the time being the problem seems related to a concurrence issue triggered by an DemandForwardingBridgeSupport#addSubscription
and an DemandForwardingBridgeSupport#onCommand invocations.

To reproduce the issue it is necessary to create a test forcing such an invocations:
# create two brokers each one with a duplex network connector to the other one
# create a generic topic and create some producers publishing on this topic. Moreover, create
some consumers consuming from this topic. Half of the producers/consumers are connected to
one broker, the other half to the other one. This scenario should trigger the DemandForwardingBridgeSupport#onCommand
invocation 
# create some specific topic with one producer and some consumers. The producer is connected
to one broker, the consumers to the other one. This scenario should trigger the  DemandForwardingBridgeSupport#addSubscription
invocation.

After few seconds after the junit start, just press the _detect deadlock_ button in jconsole.

As a side note, the deadlock shows up both with two duplex net connectors (A to B and B to
A) and with a single duplex net connector (A to B only) between the two brokers.


was (Author: matteor):
Ok, we managed to systematically reproduce the issue in a dedicated test-case (please find
it as an attachment to this issue, *AMQ5260AdvancedTest.java*). This junit must be inserted
in the _org.apache.activemq.bugs_ package within the activemq official src distro (version
5.10.0). In order to reproduce the deadlock in few seconds, the following configuration must
be applied (VMArgs for the runnable jar of the junit test launch config):
{noformat}
	-Dorg.apache.activemq.UseDedicatedTaskRunner=false 
	-Dbroker.ip.address=<broker ip goes here> 
	-Dnum.producers.spec.topic=5 
	-Dnum.consumers.spec.topic=5 
	-Dnum.producers.gen.topic=5 
	-Dnum.consumers.gen.topic=5 
	-Dnum.messages.sec=200
{noformat}

At the time being the problem seems related to a concurrence issue triggered by an DemandForwardingBridgeSupport#addSubscription
and an DemandForwardingBridgeSupport#onCommand invocations.

To reproduce the issue it is necessary to create a test forcing such an invocations:
# create two brokers each one with a duplex network connector to the other one
# create a generic topic and create some producers publishing on this topic. Moreover, create
some consumers consuming from this topic. Half of the producers/consumers are connected to
one broker, the other half to the other one. This scenario should trigger the DemandForwardingBridgeSupport#onCommand
invocation 
# create some specific topic with one producer and some consumers. The producer is connected
to one broker, the consumers to the other one. This scenario should trigger the  DemandForwardingBridgeSupport#addSubscription
invocation.

After few seconds after the junit start, just press the _detect deadlock_ button in jconsole.


> Cross talk over duplex network connection can lead to blocking (alternative take)
> ---------------------------------------------------------------------------------
>
>                 Key: AMQ-5260
>                 URL: https://issues.apache.org/jira/browse/AMQ-5260
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.9.0, 5.10.0
>            Reporter: matteo rulli
>         Attachments: AMQ5260AdvancedTest.java, AMQ_5260.patch, AMQ_5260_2.patch, AMQ_5260_3.patch,
deadlock.jpg, debug.jpg
>
>
> Pretty the same description with respect to AMQ-4328. 
> ----
> !deadlock.jpg!
> h2. Stacktraces:
> Stacktrace no.1:
> {noformat}
> Name: ActiveMQ NIO Worker 12
> State: BLOCKED on java.net.URI@1bae2b28 owned by: ActiveMQ Transport: tcp:///10.0.1.219:61616@57789
> Total blocked: 2  Total waited: 67
> Stack trace: 
>  org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:714)
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:581)
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:191)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> java.lang.Thread.run(Unknown Source)
> {noformat}
> ----
> stack trace no.2
> {noformat}
> Name: ActiveMQ Transport: tcp:///10.0.1.219:61616@57789
> State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@3cdbfa3e owned
by: ActiveMQ BrokerService[master2] Task-4
> Total blocked: 19  Total waited: 3
> Stack trace: 
>  sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source)
> java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1339)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:858)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:818)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:151)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:127)
>    - locked java.util.concurrent.atomic.AtomicBoolean@689389da
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:104)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
> org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:856)
> org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1128)
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:714)
>    - locked java.net.URI@1bae2b28
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:581)
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:191)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> org.apache.activemq.transport.failover.FailoverTransport$3.onCommand(FailoverTransport.java:196)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> java.lang.Thread.run(Unknown Source)
> {noformat}
> ----
> stack trace no.3
> {noformat}
> Name: ActiveMQ BrokerService[master2] Task-4
> State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@717bb9c owned
by: ActiveMQ Transport: tcp:///10.0.1.219:61616@57789
> Total blocked: 19  Total waited: 117
> Stack trace: 
>  sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source)
> java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:921)
> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:173)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:127)
>    - locked java.util.concurrent.atomic.AtomicBoolean@453bb109
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:104)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1339)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:858)
> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:904)
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> java.lang.Thread.run(Unknown Source)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message