Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of christian.posta@gmail.com
 designates 209.85.223.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1360010150143-4662747.post@n4.nabble.com>
References: <1360010150143-4662747.post@n4.nabble.com>
Date: Tue, 5 Feb 2013 08:01:40 -0700
Message-ID: 
 <CAKS7T5N3mJ7VJ1mc9nU4copCofz1XED8sG-Ve1pX7XH0eXxNGg@mail.gmail.com>
Subject: Re: Networked brokers dropping/suppressing messages
From: Christian Posta <christian.posta@gmail.com>
To: "users@activemq.apache.org" <users@activemq.apache.org>
Content-Type: multipart/alternative; boundary=20cf303f64eaaeecaa04d4fb7c41

--20cf303f64eaaeecaa04d4fb7c41
Content-Type: text/plain; charset=ISO-8859-1

This one seems curious. Can you post your broker configs and I can try on
my side?


On Mon, Feb 4, 2013 at 1:35 PM, photonikd <brian.nuszkowski@me.com> wrote:

> Hello All,
>
> I'm currently in the process of configuring 2 ActiveMQ (5.7.0) brokers with
> the primary objective to use persistant queues to sync messages from
> Datacenter 1, to a broker in Datacenter 2 (and vice versa) via a static
> openwire Network Connector. After building the 2 brokers and running some
> validation tests, I am experiencing sporadic behavior where some messages
> that are sent to the remote broker seem to never make it to their
> destination queue. Here is a "visual" representation of the configuration:
>
> Datacenter 1 Producer --> DC1 Broker: SYNC.OUT (Comp Q) --> DC2.SYNC (Phys
> Q)(Statically Included Network Connector)--> Datacenter Broker DC2 -->
> DC2.SYNC (Comp Q) --> SYNC.IN(Phys Q) (Consumer Attached to Q)
>
> As I mentioned perviously, the process is vice versa from DC2 back to DC1
> Datacenter 2 Producer --> DC2 Broker: SYNC.OUT (Comp Q) --> DC1.SYNC (Phys
> Q)(Statically Included Network Connector)--> Datacenter Broker DC1 -->
> DC1.SYNC (Comp Q) --> SYNC.IN(Phys Q) (Consumer Attached to Q)
>
> At first I was concerned that the messages were not actually being
> transferred at a TCP level from datacenter to datacenter as the message
> enqueue/dequeue count on the near end would increment/decrement as
> expected.
> But, the far end queue would never appear to receive the message and
> increment/decrement its queue. After enabling debug logging, I noticed that
> when the lost message behavior would occur, I would see this entry in the
> far end/receiving broker, indicating that the message did indeed arrive:
>
> suppressing duplicate message send
> [ID:brokerDC2-60716-1359583622675-0:0:1:1:272] with producerSequenceId
> [272]
> less than last stored: 328 |
> org.apache.activemq.broker.ProducerBrokerExchange | ActiveMQ Transport:
> tcp:///x.x.x.x:37877@61617
>
> While searching out other users experiencing this problem, I discovered and
> added the following configuration items to activemq.xml:
> (within the network connector configuration)
> suppressDuplicateQueueSubscriptions="false"
> suppressDuplicateTopicSubscriptions="false"
>
> (within the KahaDB configuration)
> <kahaDB maxFailoverProducersToTrack="0" />
>
> Notes
> In order to rule out Datacenter to Datacenter latency (the bandwidth
> between
> these DCs is, let's say, abundant), I built the same environment on 2
> virtual machines, locally and on the same VLAN. Regardless of the
> connectivity situation, the problem still existed in either scenario.
>
> Test Cases
> - After adding the above configuration items and restarting both ActiveMQ
> instances, the problem would still occur.
> - After adding the above configuration items, removing the existing KahaDB
> data directory, restarting the AmQ instance and letting the data directory
> re-create itself, the problem DID NOT occur again.
> - After restarting the brokers again, I was faced with this problem until I
> again cleared out the KahaDB directory.
>
>
> So, I'm left with a few unanswered questions:
> - Why am I receiving duplicate messages or why does AMQ think this is
> happening?
> - Did the configuration changes that I made (in conjunction with the Kaha
> rebuild) actually resolve the issue, if so, why?
> - The Plot Twist! Currently, I am accomplishing message replication with a
> 5.3.0 environment without issue.
>
> Any insight is greatly appreciated.
> Thanks!
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Networked-brokers-dropping-suppressing-messages-tp4662747.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>


-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

--20cf303f64eaaeecaa04d4fb7c41--