activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Duplicate messages durable subscriber
Date Tue, 04 Nov 2014 17:55:12 GMT
The functionality you're contemplating for the broker plugin seems pretty
far off from the standard ActiveMQ approach, since it involves knowing the
state of a message across the full network of brokers rather than allowing
each broker to operate as a stand-alone island that only needs to know
about itself and how to pass message to its neighbors.  So I'd be very
hesitant to go down the path you're suggesting.

And I suspect you won't have to; the functionality you're looking to use is
how ActiveMQ should work, so I think you'll find that either there's a bug
in ActiveMQ or a bug in your configuration.  Or maybe there's a minor
missing feature or one that's not been applied to all the transports it
should have; either way, I suspect the actual fix will turn out to be much
simpler than the plugin you're contemplating.

We'll see what shakes out as you simplify your configuration to remove the
less-common configuration options.  Eventually we should get to a
configuration that delivers the messages correctly but maybe doesn't have
all the configuration options enabled that you want to use, and then it's a
question of why it stops working with the less-common configuration options
you're using.

On Tue, Nov 4, 2014 at 8:16 AM, Andreas Gies <andreas@wayofquality.de>
wrote:

> Hi Tim,
>
> I will make some time tomorrow to try your suggestions.
>
> A Virtual destination would simply replace a durable subscriber with a
> Queue receiver.
> Messages posted to a mapped topic would end up in the the underlying
> queues.
>
> Now, a consumer that wasn't connected to the peer broker would immediately
> receive
> the messages on the broker - that would address the problem of temporarily
> unvisible
> messages.
>
> As for the duplicated delivery of messages - effectively we would have
> concurrent consumers
> (even though most likely onlz one of the would be connected at any one
> point in time).
> A message consumed on either side of the NWOB would effectively vanish
> from the underlying
> Queue and not be delivered again.
>
>
> Due to that I was contemplating a distributed broker plugin that would
> kind of synchronize the
> consumption of messages of a DS within a NWOB.
>
> However, before going that path I wanted to make sure that I am not
> missing something obvious.
>
>
> Thanks for the answers so far and best regards
> Andreas
>
> On 03/11/14 20:23, Tim Bain wrote:
>
>> Andreas,
>>
>> Your spec added two configuration elements your previous post didn't
>> mention, and I'd like to eliminate each of them in turn to see if it's
>> causing/contributing to the problem.
>>
>>     1. Your networkConnectors are apparently multicast.  Please see what
>>     happens if you configure them as
>>     static:(tcp://host:port?tcpOptions)?staticOptions, to take the
>> multicast
>>     (and the broker discovery that it's presumably doing) out of the
>> equation.
>>     I recently experimented with what happens when the failover is
>> allowed to
>>     perform a reconnect in a broker-to-broker networkConnector, and the
>> result
>>     is duplicate and/or stale subscriptions between the brokers.  That
>> behavior
>>     could explain what you're seeing, if multicast is similarly performing
>>     reconnects without notifying the static wrapper so it can recreate the
>>     network bridge, so let's take it out of the equation to see if the
>> behavior
>>     changes.  (I've never used multicast, so this might not make sense; if
>>     someone knows that this can't be the issue, please say so.)
>>     2. I don't know how gracefully conduitSubscriptions reacts to
>> consumers
>>
>>     moving around the network of brokers; I don't believe this should be
>> the
>>     problem, but if #1 doesn't produce any change in behavior, can you set
>>     conduitSubscriptions=false and see if anything changes?
>>
>> I'm not clear on how Virtual Topics will solve the problem; can you
>> explain?  To me this feels like a problem with broker-to-broker management
>> of subscriptions made on behalf of clients (most likely duplicate
>> subscriptions for a client, one from each broker, after a failover), and
>> I'm not sure how a Virtual Topic would make it any better if that's the
>> case.  But if you know of a way that it would, that might help me to
>> understand what's going on.
>>
>> Tim
>>
>> On Sat, Nov 1, 2014 at 9:50 AM, Andreas Gies <andreas@wayofquality.de>
>> wrote:
>>
>>  Hello Tim,
>>>
>>> thanks for your answer. It took me a bit to digest it - so my apologies
>>> for
>>> the delay in my answer.
>>>
>>> I have come up with a test case that shows - and unfortunately confirms
>>> my observations.The test case is located at [1].
>>>
>>> Here is the excerpt of my problem descriptions & observations:
>>>
>>> /**
>>>   * This specification shall help to investigate the duplicate delivery
>>> of
>>> messages for durable subscribers
>>>   * within a network of brokers. The problem has been posted on the
>>> ActiveMQ mailing list on Oct. 18th 2014
>>>   * and was described as follows:
>>>   *
>>>   * Suppose you have a network of brokers consisting of two members
>>> discovering each other via multicast.
>>>   * The network bridge is set up using conduit subscriptions. Now assume
>>> that we have a durable subscriber
>>>   * named "S" that connects to the network of brokers using a failover
>>> uri
>>> pointing to both brokers.
>>>   *
>>>   * First, the subscriber connects to Broker A. It will consume all
>>> messages published to either Broker A or B.
>>>   * Now the subscriber disconnects and stays offline for a bit, then it
>>> reconnects to Broker B. Now it will pick
>>>   * up all messages that have been published while it was offline.
>>>   *
>>>   * Let's say then 10 messages are published. All is well as the
>>> subscriber
>>> consumes those messages.
>>>   * If the subscriber then disconnects and reconnects to Broker A, these
>>> 10
>>> messages will be consumed
>>>   * again by the reconnected subscriber.
>>>   *
>>>   * According to Tim Bain on Oct., 20th 2014 this indicates a bug rather
>>> than a missing feature in ActiveMQ
>>>   * and this Spec shall pinpoint the behavior.
>>>   * *
>>>   * The test is based on ActiveMQ 5.10
>>>   *
>>>   * Observations:
>>>   * -------------
>>>   * Depending on when the durable subscriber is known to the members of
>>> the
>>> NWOB, messages can be either left pending
>>>   * or delivered repeatedly (see the last 2 test cases). Message gaps can
>>> occur, if the DS has only connected
>>>   * to one broker so far. If the DS then disconnects and after a while
>>> reconnects to the other broker it wasn't
>>>   * connected to so far, it will not see the messages that have been
>>> produced while it was offline (it will see
>>>   * those messages after reconnecting to broker 1).
>>>   *
>>>   * Dupilcate delivery will happen if the DS was already connected to
>>> both
>>> brokers. From the broker's perspective
>>>   * it seems that those DS are handled as two distinct subscribers, so
>>> effectively all messages that are published
>>>   * will eventually be delivered to both subscribers.
>>>   */
>>>
>>> I know that Virtual Topics could solve the problem - however we in the
>>> middleware team are not in control of that
>>> particular client application and therefor we cannot change the consumer
>>> from a DS to a queue consumer.
>>>
>>> Can you confirm that we are indeed looking at a missing feature or a bug
>>> in ActiveMQ 5.10 ? - Otherwise i would
>>> need to get my thinking cap back on and see how I could solve the problem
>>> without changing the client code.
>>>
>>> [1]
>>> https://github.com/woq-blended/blended/blob/master/
>>> blended-testing/blended-testing-activemq/src/test/
>>> scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
>>> <
>>> https://github.com/woq-blended/blended/blob/master/
>>> blended-testing/blended-testing-activemq/src/test/
>>> scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
>>> Thanks and best regards
>>> Andreas
>>>
>>>
>>>  On 20 Oct 2014, at 17:40, Tim Bain <tbain@alumni.duke.edu> wrote:
>>>>
>>>> If you have a network of brokers, messages on topics will be forwarded
>>>> to
>>>> whichever broker the consumer connects to, without duplicate delivery of
>>>> any messages so long as no messages were processed by the consumer
>>>>
>>> without
>>>
>>>> being ack'ed.  If you were using queues, there's the potential for
>>>>
>>> messages
>>>
>>>> to get stranded on a broker if no consumers are left, but this isn't
>>>> possible for topics.  (I'm not clear on the reason that topics can't get
>>>> messages stranded even when consumers bounce between brokers, and
>>>> unfortunately http://activemq.apache.org/networks-of-brokers.html
>>>>
>>> doesn't
>>>
>>>> describe why that is.)
>>>>
>>>> So I think that ActiveMQ's base capabilities will do exactly what you
>>>>
>>> want,
>>>
>>>> and if you're seeing redelivery of messages that were successfully acked
>>>> when the consumer bounces to another broker, I think that would indicate
>>>>
>>> a
>>>
>>>> bug in ActiveMQ rather than a missing feature.
>>>>
>>>> On Sun, Oct 19, 2014 at 6:05 PM, Noel OConnor <noel.oconnor@gmail.com>
>>>> wrote:
>>>>
>>>>  Take a look at idempotent consumers in camel. This may help you out as
>>>>> a
>>>>> basis for your plugin if you decide to go with it.
>>>>> On Oct 18, 2014 5:47 PM, "Andreas Gies" <andreas@wayofquality.de>
>>>>>
>>>> wrote:
>>>
>>>> Hi
>>>>>>
>>>>>> I am using ActiveMQ 5.10 in an application. So far the requirement
for
>>>>>>
>>>>> the
>>>>>
>>>>>> remote locations has been for pure store and forward capabilities,
>>>>>> so that a single AMQ broker was sufficient. This has changed in a
way
>>>>>>
>>>>> that
>>>>>
>>>>>> now 2 nodes should be present in the remote location for
>>>>>> resilience and load balancing. I had considered a master/configuration
>>>>>>
>>>>> as
>>>
>>>> the requirement for resilience is stronger than that for load
>>>>>>
>>>>> balancing.
>>>
>>>> However, the situation in those locations is that I don’t have a shared
>>>>>>
>>>>> db
>>>>>
>>>>>> nor a shared filesystem. As far as I have understood the replicated
>>>>>>
>>>>> level db
>>>>>
>>>>>> would require at least 3 nodes ?
>>>>>>
>>>>>> This is why I have chosen a network of brokers in the end, which
works
>>>>>> well for any Queue based communication.
>>>>>>
>>>>>> Now my problem is that there is one client application that is
>>>>>> provided
>>>>>>
>>>>> by
>>>>>
>>>>>> a 3rd party and uses durable subscriptions. It would be quite an
>>>>>> effort
>>>>>> to change that application towards using queues, so that I could
>>>>>>
>>>>> consider
>>>
>>>> virtual destinations.
>>>>>>
>>>>>> The problem occurs two-fold:
>>>>>>
>>>>>> Assume a  Subscriber connects to BrokerA, then disconnects and
>>>>>>
>>>>> reconnects
>>>
>>>> to Broker B. It consumes messages for a while, than disconnects
>>>>>> and reconnects to Broker A. All messages that have already been
>>>>>>
>>>>> consumed
>>>
>>>> while it was connected to Broker B will be delivered again.
>>>>>>
>>>>>> My question is now whether this could be avoided by means of ActiveMQ
>>>>>> alone ? - I was contemplating a broker plugin to track messages that
>>>>>> have been consumed on other nodes so that I could avoid redelivering
>>>>>>
>>>>> them
>>>
>>>> again.
>>>>>>
>>>>>> Sorry if thats a bit vague - I am fishing for ideas ….
>>>>>>
>>>>>>
>>>>>> Thanks and best regards
>>>>>> Andreas
>>>>>>
>>>>>
>>>
> --
>
>
>    Andreas Gies
>
> WoQ – Way of Quality GmbH
>
> Geschäftsführer & CTO
>
> /eMail:/andreas@wayofquality.de <mailto:andreas@wayofquality.de>
>
> /Tel:/ +49 151 23470823
>
> /Fax:/ +49 1805 006534 2114
>
> /Twitter:/ andreasgies /Skype:/ giessonic
>
> /LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/> (
> http://de.linkedin.com/pub/andreas-gies/0/594/aa5/)
>
> /Xing:/ <http://www.xing.com/profile/Andreas_Gies> (
> http://www.xing.com/profile/Andreas_Gies)
>
> /Blog:/ <http://www.wayofquality.de/index.php/en/blog> (
> http://www.wayofquality.de/index.php/en/blog)
>
> /Github:/ <https://github.com/atooni> (https://github.com/atooni)
>
> /Amtsgericht Landshut:/HRB 8352//
>
> //
>
> /Ust.-Id.:/ DE274771254
>
>
>      Haftungsausschluss
>
> Diese Email kann vertrauliche und/oder rechtlich geschützte Informationen
> enthalten und ist ausschließlich für den/die benannten Adressaten bestimmt.
> Sollten Sie nicht der beabsichtigte Empfänger sein oder diese Email
> irrtümlich erhalten haben, ist es Ihnen nicht gestattet diese Mail oder
> einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu kopieren, unbefugt
> weiterzuleiten oder zu behalten. Informieren Sie bitte sofort den Absender
> telefonisch oder per Email und löschen Sie diese Email und alle Kopien aus
> Ihrem System. Wir haften nicht für die Unversehrtheit von Emails, nachdem
> sie unseren Einflussbereich verlassen haben.
>
>
>      Disclaimer
>
> This email may contain confidential and/or privileged information and is
> intended solely for the attention and use of the named addressee(s). If you
> are not the intended recipient, or a person responsible for delivering it
> to the intended recipient, you are not authorized to and must not disclose,
> copy, distribute, or retain this message or any part of it without our
> authority. Please contact the sender by call or reply email immediately and
> destroy all copies and the original message. We are not responsible for the
> integrity of emails after they have left our sphere of control.
>
> //
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message