Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <51A36AF9.1040602@hr-group.de>
Date: Mon, 27 May 2013 16:17:29 +0200
From: Christian Wander <christian.wander@hr-group.de>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130510 Thunderbird/17.0.6
MIME-Version: 1.0
To: <users@activemq.apache.org>
Subject: Vanishing NetworkBridges with unreliable WAN connections
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit

Hi users,

we've deployed ActiveMQ 5.7 on several remote hosts and configured a
static broker network between the remote nodes and a central broker in
our datacenter according to [1] (typical star architecture).

The setup seems to be working fine, messages enqueued on the remote
brokers get forwarded to the central broker via the network bridge.
If the underlying WAN connection is interrupted for some reason, the
bridge is able to recover the connection to the central broker most of
the time.

Sometimes though the remote broker can't recover the connection and
messages are starting to pile up. Around the same time the last message
is forwarded to the central broker, the log file of the remote broker
shows that it's trying to establish the network bridge with the central
broker.

This attempt fails with the message: "Failed to add static destination
queue://queue1" and a TransportDisposedIOException (see [3] for the log 
entry).

 From this moment on the network bridge to the central broker won't be
established again. It seems that this situation can only be resolved
by restarting the remote broker process.

Has anybody experienced this behavior before with unreliable WAN
connections?

Any help would be appreciated.
Chris

[1] http://activemq.apache.org/networks-of-brokers.html

[2] Configuration of network bridge on the remote broker

<networkConnectors>
      <networkConnector
         uri="static:(tcp://centralbroker:61616)"
         staticBridge="true"
         name="bridgeToDataCenter"
         duplex="false"
         conduitSubscriptions="true" >

         <staticallyIncludedDestinations>
             <queue physicalName="queue1" />
         </staticallyIncludedDestinations>

      </networkConnector>
</networkConnectors>

[3] activemq.log

2013-05-22 17:30:30,847 | INFO  | il00347#94 | 
mq.network.DemandForwardingBridgeSupport | Network connection between 
vm://remotebroker#94 and 
tcp://centralbroker/1.2.3.4:61616@42764(centralbroker) has been established.
2013-05-22 17:30:30,913 | WARN  | 1616@42764 | 
mq.network.DemandForwardingBridgeSupport | Network connection between 
vm://remotebroker#94 and tcp://centralbroker/1.2.3.4:61616@42764 
shutdown due to a remote error: javax.jms.InvalidClientIDException: 
Broker: centralbroker - Client: 
remotebroker-queues_remotebroker_outbound already connected from 
tcp://4.3.2.1:42202
2013-05-22 17:30:31,021 | INFO  | 7] Task-78 | 
mq.network.DemandForwardingBridgeSupport | remotebroker bridge to 
centralbroker stopped
2013-05-22 17:30:31,023 | ERROR | il00347#94 | 
mq.network.DemandForwardingBridgeSupport | Failed to add static 
destination queue://queue1
org.apache.activemq.transport.TransportDisposedIOException: Peer 
(vm://remotebroker#95) disposed.
         at 
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:89)
         at 
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
         at 
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
         at 
org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:649)
         at 
org.apache.activemq.network.DemandForwardingBridgeSupport.setupStaticDestinations(DemandForwardingBridgeSupport.java:981)
         at 
org.apache.activemq.network.DurableConduitBridge.setupStaticDestinations(DurableConduitBridge.java:53)
         at 
org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:281)
         at 
org.apache.activemq.network.DemandForwardingBridgeSupport.access$000(DemandForwardingBridgeSupport.java:92)
         at 
org.apache.activemq.network.DemandForwardingBridgeSupport$3.run(DemandForwardingBridgeSupport.java:209)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)