activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Van Dewoestine <ervan...@gmail.com>
Subject Re: Issue creating a distributed queue using store and forward.
Date Thu, 08 Oct 2009 14:39:03 GMT
I tried setting the prefetchSize to 1 for both consumers, but
unfortunately, that didn't resolve the issue.  Messages still end up
stuck on B2 when C2 is stopped.

On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote:
> set the prefetchSize=1 on your network - to avoid orphaned messages on
> B2
>
> On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
>
> > Hmm, that seems like a flawed approach if the broker it visited has
> > changed state and is now eligible to process those messages.  How is
> > this type of fail over intended to be handled with activemq?  Is there
> > an alternate solution that I'm missing?
> >
> > On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
> >>
> >> I think that by design a message will not be forwarded to a broker
> >> that it
> >> has already visited.
> >>
> >> Joe
> >> http://www.ttmsolutions.com
> >>
> >>
> >> Eric Van wrote:
> >>>
> >>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
> >>>
> >>> I'm running into an issue with the store and forward feature of
> >>> activemq, which I using in an attempt to create a highly available
> >>> distributed queue.  I'm trying to figure out if the issue is a
> >>> misconfiguration on my part, expected behavior of activemq, or a
> >>> bug.
> >>>
> >>> The summary of the problem is that given 2 brokers, B1 and B2, which
> >>> each have one consumer, C1 and C2, which are subscribed to the same
> >>> queue.  If I stop a consumer on one of the brokers, the pending
> >>> messages from that broker are not always forwarded to the other
> >>> broker
> >>> which still has a consumer, leading to those messages getting
> >>> indefinitely stuck.
> >>>
> >>> The steps I use to reproduce this scenario are as follows (Note: all
> >>> producing and consuming is performed over the stomp transport):
> >>>
> >>> Couple notes about the consumers:
> >>> - they have a prefetchSize of 40
> >>> - the processing of messages can take some time, so for the purposes
> >>>  of this exercise, I've created a simple consumer that sleeps for 10
> >>>  seconds before sending the message ack (using client-individual ack
> >>>  mode)
> >>>
> >>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
> >>>   yet running.
> >>> 2. produce a few thousand messages to B1
> >>>   Note: B1 now has a few thousand pending messages and B2 has 0.
> >>> 3. start consumer C2 (listing for messages from B2)
> >>>   Note: messages are are successfully received and begin processing
> >>>   (monitoring the brokers shows pending messages decreasing). Now B2
> >>>   has all the pending messages and B1 has 0.
> >>> 4. start consumer C1 (listing for messages from B1)
> >>>   Note: no messages are received, which is another issue I have
> >>> since
> >>>   B2 now has thousands of pending messages which C1 could help
> >>>   process, but instead sits idle while C2 is forced to handle all
> >>> the
> >>>   messages.
> >>> 5. stop consumer C2
> >>>   Note: now I have thousands of messages sitting on B2 and 0 on B1
> >>>   where a C1 is alive and ready to handle them.  So at this point,
> >>>   despite having a consumer running, thousands of messages are stuck
> >>>   in the queue.
> >>> 6. stop consumer C1
> >>>   Note: now I have no consumers.  Stopping and restarting C1 has no
> >>>   effect on the pending messages sitting on B1's queue.
> >>> 7. stop both brokers
> >>> 8. start B1, then start B2
> >>> 9. start C1
> >>>   Note: now all messages have migrated from B2 to B1 and C1 is again
> >>>   processing messages.
> >>>
> >>> So after step 5, the only way to recover from the stuck messages
> >>> is to
> >>> restart the brokers.
> >>>
> >>> Below is my current connector config which I have on both brokers.
> >>> I've tried playing with the various properties of the connector, but
> >>> it seems as though no matter what I try the above scenario continues
> >>> to occur.
> >>>
> >>> <networkConnector
> >>>    name="default-nc"
> >>>    uri="multicast://default?group=${broker.group}"
> >>>    dynamicOnly="true"
> >>>    networkTTL="25"
> >>>    suppressDuplicateQueueSubscriptions="true"/>
> >>>
> >>>
> >>> So, is this an activemq bug? Am I mis-using activemq? Is there some
> >>> other way to achieve a highly available distributed queue?
> >>>
> >>> Any help in this regard is greatly appreciated.
> >>>
> >>> --
> >>> eric
> >>>
> >>>
> >
> > --
> > eric
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/

--
eric

Mime
View raw message