activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <>
Subject Re: Issue creating a distributed queue using store and forward.
Date Thu, 08 Oct 2009 08:33:06 GMT
set the prefetchSize=1 on your network - to avoid orphaned messages on  

On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:

> Hmm, that seems like a flawed approach if the broker it visited has
> changed state and is now eligible to process those messages.  How is
> this type of fail over intended to be handled with activemq?  Is there
> an alternate solution that I'm missing?
> On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
>> I think that by design a message will not be forwarded to a broker  
>> that it
>> has already visited.
>> Joe
>> Eric Van wrote:
>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
>>> I'm running into an issue with the store and forward feature of
>>> activemq, which I using in an attempt to create a highly available
>>> distributed queue.  I'm trying to figure out if the issue is a
>>> misconfiguration on my part, expected behavior of activemq, or a  
>>> bug.
>>> The summary of the problem is that given 2 brokers, B1 and B2, which
>>> each have one consumer, C1 and C2, which are subscribed to the same
>>> queue.  If I stop a consumer on one of the brokers, the pending
>>> messages from that broker are not always forwarded to the other  
>>> broker
>>> which still has a consumer, leading to those messages getting
>>> indefinitely stuck.
>>> The steps I use to reproduce this scenario are as follows (Note: all
>>> producing and consuming is performed over the stomp transport):
>>> Couple notes about the consumers:
>>> - they have a prefetchSize of 40
>>> - the processing of messages can take some time, so for the purposes
>>>  of this exercise, I've created a simple consumer that sleeps for 10
>>>  seconds before sending the message ack (using client-individual ack
>>>  mode)
>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
>>>   yet running.
>>> 2. produce a few thousand messages to B1
>>>   Note: B1 now has a few thousand pending messages and B2 has 0.
>>> 3. start consumer C2 (listing for messages from B2)
>>>   Note: messages are are successfully received and begin processing
>>>   (monitoring the brokers shows pending messages decreasing). Now B2
>>>   has all the pending messages and B1 has 0.
>>> 4. start consumer C1 (listing for messages from B1)
>>>   Note: no messages are received, which is another issue I have  
>>> since
>>>   B2 now has thousands of pending messages which C1 could help
>>>   process, but instead sits idle while C2 is forced to handle all  
>>> the
>>>   messages.
>>> 5. stop consumer C2
>>>   Note: now I have thousands of messages sitting on B2 and 0 on B1
>>>   where a C1 is alive and ready to handle them.  So at this point,
>>>   despite having a consumer running, thousands of messages are stuck
>>>   in the queue.
>>> 6. stop consumer C1
>>>   Note: now I have no consumers.  Stopping and restarting C1 has no
>>>   effect on the pending messages sitting on B1's queue.
>>> 7. stop both brokers
>>> 8. start B1, then start B2
>>> 9. start C1
>>>   Note: now all messages have migrated from B2 to B1 and C1 is again
>>>   processing messages.
>>> So after step 5, the only way to recover from the stuck messages  
>>> is to
>>> restart the brokers.
>>> Below is my current connector config which I have on both brokers.
>>> I've tried playing with the various properties of the connector, but
>>> it seems as though no matter what I try the above scenario continues
>>> to occur.
>>> <networkConnector
>>>    name="default-nc"
>>>    uri="multicast://default?group=${}"
>>>    dynamicOnly="true"
>>>    networkTTL="25"
>>>    suppressDuplicateQueueSubscriptions="true"/>
>>> So, is this an activemq bug? Am I mis-using activemq? Is there some
>>> other way to achieve a highly available distributed queue?
>>> Any help in this regard is greatly appreciated.
>>> --
>>> eric
> --
> eric

Rob Davies
I work here:
My Blog:
I'm writing this:

View raw message