activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Gies <andr...@wayofquality.de>
Subject Re: Duplicate messages durable subscriber
Date Sat, 01 Nov 2014 15:50:21 GMT
Hello Tim,

thanks for your answer. It took me a bit to digest it - so my apologies for 
the delay in my answer. 

I have come up with a test case that shows - and unfortunately confirms 
my observations.The test case is located at [1]. 

Here is the excerpt of my problem descriptions & observations:

/**
 * This specification shall help to investigate the duplicate delivery of messages for durable
subscribers
 * within a network of brokers. The problem has been posted on the ActiveMQ mailing list on
Oct. 18th 2014
 * and was described as follows:
 *
 * Suppose you have a network of brokers consisting of two members discovering each other
via multicast.
 * The network bridge is set up using conduit subscriptions. Now assume that we have a durable
subscriber
 * named "S" that connects to the network of brokers using a failover uri pointing to both
brokers.
 *
 * First, the subscriber connects to Broker A. It will consume all messages published to either
Broker A or B.
 * Now the subscriber disconnects and stays offline for a bit, then it reconnects to Broker
B. Now it will pick
 * up all messages that have been published while it was offline.
 *
 * Let's say then 10 messages are published. All is well as the subscriber consumes those
messages.
 * If the subscriber then disconnects and reconnects to Broker A, these 10 messages will be
consumed
 * again by the reconnected subscriber. 
 *
 * According to Tim Bain on Oct., 20th 2014 this indicates a bug rather than a missing feature
in ActiveMQ
 * and this Spec shall pinpoint the behavior.
 * *
 * The test is based on ActiveMQ 5.10
 *
 * Observations:
 * -------------
 * Depending on when the durable subscriber is known to the members of the NWOB, messages
can be either left pending
 * or delivered repeatedly (see the last 2 test cases). Message gaps can occur, if the DS
has only connected
 * to one broker so far. If the DS then disconnects and after a while reconnects to the other
broker it wasn't
 * connected to so far, it will not see the messages that have been produced while it was
offline (it will see
 * those messages after reconnecting to broker 1).
 *
 * Dupilcate delivery will happen if the DS was already connected to both brokers. From the
broker's perspective
 * it seems that those DS are handled as two distinct subscribers, so effectively all messages
that are published
 * will eventually be delivered to both subscribers.
 */

I know that Virtual Topics could solve the problem - however we in the middleware team are
not in control of that
particular client application and therefor we cannot change the consumer from a DS to a queue
consumer. 

Can you confirm that we are indeed looking at a missing feature or a bug in ActiveMQ 5.10
? - Otherwise i would 
need to get my thinking cap back on and see how I could solve the problem without changing
the client code. 

[1] https://github.com/woq-blended/blended/blob/master/blended-testing/blended-testing-activemq/src/test/scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
<https://github.com/woq-blended/blended/blob/master/blended-testing/blended-testing-activemq/src/test/scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala>

Thanks and best regards
Andreas


> On 20 Oct 2014, at 17:40, Tim Bain <tbain@alumni.duke.edu> wrote:
> 
> If you have a network of brokers, messages on topics will be forwarded to
> whichever broker the consumer connects to, without duplicate delivery of
> any messages so long as no messages were processed by the consumer without
> being ack'ed.  If you were using queues, there's the potential for messages
> to get stranded on a broker if no consumers are left, but this isn't
> possible for topics.  (I'm not clear on the reason that topics can't get
> messages stranded even when consumers bounce between brokers, and
> unfortunately http://activemq.apache.org/networks-of-brokers.html doesn't
> describe why that is.)
> 
> So I think that ActiveMQ's base capabilities will do exactly what you want,
> and if you're seeing redelivery of messages that were successfully acked
> when the consumer bounces to another broker, I think that would indicate a
> bug in ActiveMQ rather than a missing feature.
> 
> On Sun, Oct 19, 2014 at 6:05 PM, Noel OConnor <noel.oconnor@gmail.com>
> wrote:
> 
>> Take a look at idempotent consumers in camel. This may help you out as a
>> basis for your plugin if you decide to go with it.
>> On Oct 18, 2014 5:47 PM, "Andreas Gies" <andreas@wayofquality.de> wrote:
>> 
>>> Hi
>>> 
>>> I am using ActiveMQ 5.10 in an application. So far the requirement for
>> the
>>> remote locations has been for pure store and forward capabilities,
>>> so that a single AMQ broker was sufficient. This has changed in a way
>> that
>>> now 2 nodes should be present in the remote location for
>>> resilience and load balancing. I had considered a master/configuration as
>>> the requirement for resilience is stronger than that for load balancing.
>>> 
>>> However, the situation in those locations is that I don’t have a shared
>> db
>>> nor a shared filesystem. As far as I have understood the replicated
>> level db
>>> would require at least 3 nodes ?
>>> 
>>> This is why I have chosen a network of brokers in the end, which works
>>> well for any Queue based communication.
>>> 
>>> Now my problem is that there is one client application that is provided
>> by
>>> a 3rd party and uses durable subscriptions. It would be quite an effort
>>> to change that application towards using queues, so that I could consider
>>> virtual destinations.
>>> 
>>> The problem occurs two-fold:
>>> 
>>> Assume a  Subscriber connects to BrokerA, then disconnects and reconnects
>>> to Broker B. It consumes messages for a while, than disconnects
>>> and reconnects to Broker A. All messages that have already been consumed
>>> while it was connected to Broker B will be delivered again.
>>> 
>>> My question is now whether this could be avoided by means of ActiveMQ
>>> alone ? - I was contemplating a broker plugin to track messages that
>>> have been consumed on other nodes so that I could avoid redelivering them
>>> again.
>>> 
>>> Sorry if thats a bit vague - I am fishing for ideas ….
>>> 
>>> 
>>> Thanks and best regards
>>> Andreas
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message