activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Van Dewoestine <ervan...@gmail.com>
Subject Re: Issue creating a distributed queue using store and forward.
Date Thu, 08 Oct 2009 17:32:10 GMT
Ah, that attribute isn't documented on the activemq site ;)

I tried that one out, but unfortunately I don't see any change in
behavior.  I verified that the prefetchSize is indeed set to 1 by
examining the network connector attributes for both brokers via jmx.

Thank you so much for your help so far, I really do appreciate it.

I'm going to start digging through the code to see if I can determine
why these messages refuse to budge.  I'll probably start by seeing if
Joe's assumption about messages not visiting a broker more than once
is the cause.

If you or anyone else has any other tips or suggestions, I'd love to
hear them.

On Thu, Oct 08, 2009 at 05:42:48PM +0100, Rob Davies wrote:
> Hi Eric,
>
> sorry for being so quick in my reply - in the broker xml config set
> the prefetchSize property on the network - e.g.
>
> <networkConnectors>
>          <networkConnector
>               uri="static://(tcp://localhost:61617)"
>               name="Connection to 61617"
>               dynamicOnly="true"
> 	     prefetchSize="1"/>
>
>   </networkConnectors>
> On 8 Oct 2009, at 17:16, Eric Van Dewoestine wrote:
>
> > I guess I'm not sure what setting you are referring to.  I tried
> > setting a couple different prefetch policies on the tcp connector like
> > so:
> >
> > 1st attempt:
> > <transportConnector
> >  uri="tcp://${broker.host}:0?jms.prefetchPolicy.queuePrefetch=1"
> >  discoveryUri="multicast://default?group=${broker.group}"/>
> >
> > 2nd attempt:
> > <transportConnector
> >  uri="tcp://${broker.host}:0?jms.prefetchPolicy.all=1"
> >  discoveryUri="multicast://default?group=${broker.group}"/>
> >
> > But neither of them seemed to change the behavior.  Are you referring
> > to another setting?  Can you provide an example of the setting you are
> > referring to?
> >
> > On Thu, Oct 08, 2009 at 04:22:11PM +0100, Rob Davies wrote:
> >> do you set the prefetchSize property on the network itself ? Setting
> >> it on the consumers wouldn't make any difference
> >> On 8 Oct 2009, at 15:39, Eric Van Dewoestine wrote:
> >>
> >>> I tried setting the prefetchSize to 1 for both consumers, but
> >>> unfortunately, that didn't resolve the issue.  Messages still end up
> >>> stuck on B2 when C2 is stopped.
> >>>
> >>> On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote:
> >>>> set the prefetchSize=1 on your network - to avoid orphaned messages
> >>>> on
> >>>> B2
> >>>>
> >>>> On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
> >>>>
> >>>>> Hmm, that seems like a flawed approach if the broker it visited
> >>>>> has
> >>>>> changed state and is now eligible to process those messages.
> >>>>> How is
> >>>>> this type of fail over intended to be handled with activemq?  Is
> >>>>> there
> >>>>> an alternate solution that I'm missing?
> >>>>>
> >>>>> On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
> >>>>>>
> >>>>>> I think that by design a message will not be forwarded to a
> >>>>>> broker
> >>>>>> that it
> >>>>>> has already visited.
> >>>>>>
> >>>>>> Joe
> >>>>>> http://www.ttmsolutions.com
> >>>>>>
> >>>>>>
> >>>>>> Eric Van wrote:
> >>>>>>>
> >>>>>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots
> >>>>>>> listing)
> >>>>>>>
> >>>>>>> I'm running into an issue with the store and forward feature
of
> >>>>>>> activemq, which I using in an attempt to create a highly
> >>>>>>> available
> >>>>>>> distributed queue.  I'm trying to figure out if the issue
is a
> >>>>>>> misconfiguration on my part, expected behavior of activemq,
or a
> >>>>>>> bug.
> >>>>>>>
> >>>>>>> The summary of the problem is that given 2 brokers, B1 and
B2,
> >>>>>>> which
> >>>>>>> each have one consumer, C1 and C2, which are subscribed
to the
> >>>>>>> same
> >>>>>>> queue.  If I stop a consumer on one of the brokers, the
pending
> >>>>>>> messages from that broker are not always forwarded to the
other
> >>>>>>> broker
> >>>>>>> which still has a consumer, leading to those messages getting
> >>>>>>> indefinitely stuck.
> >>>>>>>
> >>>>>>> The steps I use to reproduce this scenario are as follows
(Note:
> >>>>>>> all
> >>>>>>> producing and consuming is performed over the stomp transport):
> >>>>>>>
> >>>>>>> Couple notes about the consumers:
> >>>>>>> - they have a prefetchSize of 40
> >>>>>>> - the processing of messages can take some time, so for
the
> >>>>>>> purposes
> >>>>>>> of this exercise, I've created a simple consumer that sleeps
for
> >>>>>>> 10
> >>>>>>> seconds before sending the message ack (using client-individual
> >>>>>>> ack
> >>>>>>> mode)
> >>>>>>>
> >>>>>>> 1. start both brokers (B1 and B2). The consumers (C1 and
C2) are
> >>>>>>> not
> >>>>>>> yet running.
> >>>>>>> 2. produce a few thousand messages to B1
> >>>>>>> Note: B1 now has a few thousand pending messages and B2
has 0.
> >>>>>>> 3. start consumer C2 (listing for messages from B2)
> >>>>>>> Note: messages are are successfully received and begin
> >>>>>>> processing
> >>>>>>> (monitoring the brokers shows pending messages decreasing).
Now
> >>>>>>> B2
> >>>>>>> has all the pending messages and B1 has 0.
> >>>>>>> 4. start consumer C1 (listing for messages from B1)
> >>>>>>> Note: no messages are received, which is another issue I
have
> >>>>>>> since
> >>>>>>> B2 now has thousands of pending messages which C1 could
help
> >>>>>>> process, but instead sits idle while C2 is forced to handle
all
> >>>>>>> the
> >>>>>>> messages.
> >>>>>>> 5. stop consumer C2
> >>>>>>> Note: now I have thousands of messages sitting on B2 and
0 on B1
> >>>>>>> where a C1 is alive and ready to handle them.  So at this
point,
> >>>>>>> despite having a consumer running, thousands of messages
are
> >>>>>>> stuck
> >>>>>>> in the queue.
> >>>>>>> 6. stop consumer C1
> >>>>>>> Note: now I have no consumers.  Stopping and restarting
C1 has
> >>>>>>> no
> >>>>>>> effect on the pending messages sitting on B1's queue.
> >>>>>>> 7. stop both brokers
> >>>>>>> 8. start B1, then start B2
> >>>>>>> 9. start C1
> >>>>>>> Note: now all messages have migrated from B2 to B1 and C1
is
> >>>>>>> again
> >>>>>>> processing messages.
> >>>>>>>
> >>>>>>> So after step 5, the only way to recover from the stuck
messages
> >>>>>>> is to
> >>>>>>> restart the brokers.
> >>>>>>>
> >>>>>>> Below is my current connector config which I have on both
> >>>>>>> brokers.
> >>>>>>> I've tried playing with the various properties of the connector,
> >>>>>>> but
> >>>>>>> it seems as though no matter what I try the above scenario
> >>>>>>> continues
> >>>>>>> to occur.
> >>>>>>>
> >>>>>>> <networkConnector
> >>>>>>>  name="default-nc"
> >>>>>>>  uri="multicast://default?group=${broker.group}"
> >>>>>>>  dynamicOnly="true"
> >>>>>>>  networkTTL="25"
> >>>>>>>  suppressDuplicateQueueSubscriptions="true"/>
> >>>>>>>
> >>>>>>>
> >>>>>>> So, is this an activemq bug? Am I mis-using activemq? Is
there
> >>>>>>> some
> >>>>>>> other way to achieve a highly available distributed queue?
> >>>>>>>
> >>>>>>> Any help in this regard is greatly appreciated.
> >>>>>>>
> >>>>>>> --
> >>>>>>> eric
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>> eric
> >>>>
> >>>> Rob Davies
> >>>> http://twitter.com/rajdavies
> >>>> I work here: http://fusesource.com
> >>>> My Blog: http://rajdavies.blogspot.com/
> >>>> I'm writing this: http://www.manning.com/snyder/
> >>>
> >>> --
> >>> eric
> >>
> >> Rob Davies
> >> http://twitter.com/rajdavies
> >> I work here: http://fusesource.com
> >> My Blog: http://rajdavies.blogspot.com/
> >> I'm writing this: http://www.manning.com/snyder/
> >>
> >>
> >>
> >>
> >>
> >
> > --
> > eric
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>

--
eric

Mime
View raw message