activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ranjit Nethi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMQ-6419) Durable Subscribers going offline - Deadlock - Network Of Brokers
Date Tue, 06 Sep 2016 20:30:21 GMT
Ranjit Nethi created AMQ-6419:
---------------------------------

             Summary: Durable Subscribers going offline - Deadlock - Network Of Brokers
                 Key: AMQ-6419
                 URL: https://issues.apache.org/jira/browse/AMQ-6419
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.10.0
         Environment: Linux - Centos 5.8
Active MQ Version 5.10.0
Container - Karaf 3.0.3
Camel 2.14.1
            Reporter: Ranjit Nethi


We are using ActiveMQ, using Network of Brokers in Hub-Spoke topology.
Active MQ is deployed in Karaf Container

Software Stack Used
Active MQ Version 5.10.0
Container - Karaf 3.0.3
Camel 2.14.1

Hub is having 2 brokers- Master Active MQ and Slave Active MQ, configured using shared file
system (GFS) and Kaha DB Data Store. Operate as failover. Spokes or Site connect to Hub using
network connectors configuration below-

<networkConnectors>
<networkConnector
  name="T:broker1->broker2"
  uri="masterslave:(tcp://masterbroker.com:61616,tcp://slavebroker.com:61616)
  duplex="true"
  decreaseNetworkConsumerPriority="true"
  networkTTL="2"
  dynamicOnly="true">
</networkConnector>

<networkConnector
  name="Q:broker1->broker2"
  uri="masterslave:(tcp://masterbroker.com:61616,tcp://slavebroker.com:61616)
  duplex="true"
  decreaseNetworkConsumerPriority="true"
  networkTTL="2"
  dynamicOnly="true">
</networkConnector>
</networkConnectors>

Issue


We have topic on HUB AMQ broker called Shop.Canonical.WorkOrder.Notify, and the producers
are at the spoke level. The producers write to the local topic Shop.Canonical.WorkOrder.Notify
(This is the same topic AMQ creates when using network of brokers).

We have Durable Topic Subscribers on Central (HUB ) broker, camel client routes subscribed
to this topic.
Apparently due to network connections lost or force shutdowns of servers hosting these karaf
containers at spoke level, we are ending up having offline durable subscribers.

It looks like we had network outage and network failover tried to stop the network bridge
and at the same time active mq tried to start another connection causing deadlock. I see the
following exception from log file. Also the details of the threads causing deadlock towards
the end of this message.



javax.jms.InvalidClientIDException: Broker: taxxx-amq-broker - Client: T:broker1->broker2_central-amq-broker_inbound_taxxx-amq-broker
already connected from vm://taxxx-amq-broker#4548
        at org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:246)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.broker.jmx.ManagedRegionBroker.addConnection(ManagedRegionBroker.java:231)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:89)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.security.JaasAuthenticationBroker.addConnection(JaasAuthenticationBroker.java:87)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:764)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:453)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.network.DemandForwardingBridgeSupport.doStartLocalAndRemoteBridges(DemandForwardingBridgeSupport.java:420)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.network.DemandForwardingBridgeSupport.access$500(DemandForwardingBridgeSupport.java:105)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at org.apache.activemq.network.DemandForwardingBridgeSupport$5.run(DemandForwardingBridgeSupport.java:335)[83:org.apache.activemq.activemq-osgi:5.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_75]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_75]
        at java.lang.Thread.run(Thread.java:745)[:1.7.0_75]




Above scenario also creates a dead lock condition - 3 threads

Thread 1 - Name: ActiveMQ BrokerService[taxxx-amq-broker] Task-17676
State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@138f39e owned by: ActiveMQ
Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480
Total blocked: 5  Total waited: 10

Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:970)
org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:206)
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)
   - locked java.util.concurrent.atomic.AtomicBoolean@1e5f8fb
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1370)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:889)
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:935)
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

 Thread 2 -
Name: ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480
State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@1c83e84 owned by: ActiveMQ
BrokerService[taxxx-amq-broker] Task-17676
Total blocked: 0  Total waited: 2

Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1370)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:889)
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:849)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)
   - locked java.util.concurrent.atomic.AtomicBoolean@12796af
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:905)
org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1178)
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:763)
   - locked java.net.URI@15e6d6
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:614)
org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
org.apache.activemq.transport.failover.FailoverTransport$3.onCommand(FailoverTransport.java:208)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
java.lang.Thread.run(Thread.java:745)



Thread 3 - Name: ActiveMQ BrokerService[taxxx-amq-broker] Task-18436
State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@138f39e owned by: ActiveMQ
Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480
Total blocked: 0  Total waited: 44

Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:288)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message