activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Networked brokers dropping/suppressing messages
Date Tue, 05 Feb 2013 15:01:40 GMT
This one seems curious. Can you post your broker configs and I can try on
my side?


On Mon, Feb 4, 2013 at 1:35 PM, photonikd <brian.nuszkowski@me.com> wrote:

> Hello All,
>
> I'm currently in the process of configuring 2 ActiveMQ (5.7.0) brokers with
> the primary objective to use persistant queues to sync messages from
> Datacenter 1, to a broker in Datacenter 2 (and vice versa) via a static
> openwire Network Connector. After building the 2 brokers and running some
> validation tests, I am experiencing sporadic behavior where some messages
> that are sent to the remote broker seem to never make it to their
> destination queue. Here is a "visual" representation of the configuration:
>
> Datacenter 1 Producer --> DC1 Broker: SYNC.OUT (Comp Q) --> DC2.SYNC (Phys
> Q)(Statically Included Network Connector)--> Datacenter Broker DC2 -->
> DC2.SYNC (Comp Q) --> SYNC.IN(Phys Q) (Consumer Attached to Q)
>
> As I mentioned perviously, the process is vice versa from DC2 back to DC1
> Datacenter 2 Producer --> DC2 Broker: SYNC.OUT (Comp Q) --> DC1.SYNC (Phys
> Q)(Statically Included Network Connector)--> Datacenter Broker DC1 -->
> DC1.SYNC (Comp Q) --> SYNC.IN(Phys Q) (Consumer Attached to Q)
>
> At first I was concerned that the messages were not actually being
> transferred at a TCP level from datacenter to datacenter as the message
> enqueue/dequeue count on the near end would increment/decrement as
> expected.
> But, the far end queue would never appear to receive the message and
> increment/decrement its queue. After enabling debug logging, I noticed that
> when the lost message behavior would occur, I would see this entry in the
> far end/receiving broker, indicating that the message did indeed arrive:
>
> suppressing duplicate message send
> [ID:brokerDC2-60716-1359583622675-0:0:1:1:272] with producerSequenceId
> [272]
> less than last stored: 328 |
> org.apache.activemq.broker.ProducerBrokerExchange | ActiveMQ Transport:
> tcp:///x.x.x.x:37877@61617
>
> While searching out other users experiencing this problem, I discovered and
> added the following configuration items to activemq.xml:
> (within the network connector configuration)
> suppressDuplicateQueueSubscriptions="false"
> suppressDuplicateTopicSubscriptions="false"
>
> (within the KahaDB configuration)
> <kahaDB maxFailoverProducersToTrack="0" />
>
> Notes
> In order to rule out Datacenter to Datacenter latency (the bandwidth
> between
> these DCs is, let's say, abundant), I built the same environment on 2
> virtual machines, locally and on the same VLAN. Regardless of the
> connectivity situation, the problem still existed in either scenario.
>
> Test Cases
> - After adding the above configuration items and restarting both ActiveMQ
> instances, the problem would still occur.
> - After adding the above configuration items, removing the existing KahaDB
> data directory, restarting the AmQ instance and letting the data directory
> re-create itself, the problem DID NOT occur again.
> - After restarting the brokers again, I was faced with this problem until I
> again cleared out the KahaDB directory.
>
>
> So, I'm left with a few unanswered questions:
> - Why am I receiving duplicate messages or why does AMQ think this is
> happening?
> - Did the configuration changes that I made (in conjunction with the Kaha
> rebuild) actually resolve the issue, if so, why?
> - The Plot Twist! Currently, I am accomplishing message replication with a
> 5.3.0 environment without issue.
>
> Any insight is greatly appreciated.
> Thanks!
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Networked-brokers-dropping-suppressing-messages-tp4662747.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message