activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From photonikd <brian.nuszkow...@me.com>
Subject Networked brokers dropping/suppressing messages
Date Mon, 04 Feb 2013 20:35:50 GMT
Hello All, 

I'm currently in the process of configuring 2 ActiveMQ (5.7.0) brokers with
the primary objective to use persistant queues to sync messages from
Datacenter 1, to a broker in Datacenter 2 (and vice versa) via a static
openwire Network Connector. After building the 2 brokers and running some
validation tests, I am experiencing sporadic behavior where some messages
that are sent to the remote broker seem to never make it to their
destination queue. Here is a "visual" representation of the configuration: 

Datacenter 1 Producer --> DC1 Broker: SYNC.OUT (Comp Q) --> DC2.SYNC (Phys
Q)(Statically Included Network Connector)--> Datacenter Broker DC2 -->
DC2.SYNC (Comp Q) --> SYNC.IN(Phys Q) (Consumer Attached to Q) 

As I mentioned perviously, the process is vice versa from DC2 back to DC1 
Datacenter 2 Producer --> DC2 Broker: SYNC.OUT (Comp Q) --> DC1.SYNC (Phys
Q)(Statically Included Network Connector)--> Datacenter Broker DC1 -->
DC1.SYNC (Comp Q) --> SYNC.IN(Phys Q) (Consumer Attached to Q) 

At first I was concerned that the messages were not actually being
transferred at a TCP level from datacenter to datacenter as the message
enqueue/dequeue count on the near end would increment/decrement as expected.
But, the far end queue would never appear to receive the message and
increment/decrement its queue. After enabling debug logging, I noticed that
when the lost message behavior would occur, I would see this entry in the
far end/receiving broker, indicating that the message did indeed arrive: 

suppressing duplicate message send
[ID:brokerDC2-60716-1359583622675-0:0:1:1:272] with producerSequenceId [272]
less than last stored: 328 |
org.apache.activemq.broker.ProducerBrokerExchange | ActiveMQ Transport:
tcp:///x.x.x.x:37877@61617 

While searching out other users experiencing this problem, I discovered and
added the following configuration items to activemq.xml: 
(within the network connector configuration) 
suppressDuplicateQueueSubscriptions="false" 
suppressDuplicateTopicSubscriptions="false" 

(within the KahaDB configuration) 
<kahaDB maxFailoverProducersToTrack="0" />

Notes
In order to rule out Datacenter to Datacenter latency (the bandwidth between
these DCs is, let's say, abundant), I built the same environment on 2
virtual machines, locally and on the same VLAN. Regardless of the
connectivity situation, the problem still existed in either scenario. 

Test Cases
- After adding the above configuration items and restarting both ActiveMQ
instances, the problem would still occur. 
- After adding the above configuration items, removing the existing KahaDB
data directory, restarting the AmQ instance and letting the data directory
re-create itself, the problem DID NOT occur again.
- After restarting the brokers again, I was faced with this problem until I
again cleared out the KahaDB directory. 


So, I'm left with a few unanswered questions: 
- Why am I receiving duplicate messages or why does AMQ think this is
happening? 
- Did the configuration changes that I made (in conjunction with the Kaha
rebuild) actually resolve the issue, if so, why? 
- The Plot Twist! Currently, I am accomplishing message replication with a
5.3.0 environment without issue. 

Any insight is greatly appreciated. 
Thanks! 




--
View this message in context: http://activemq.2283324.n4.nabble.com/Networked-brokers-dropping-suppressing-messages-tp4662747.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message