Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of rajdavies@gmail.com designates
 74.125.92.147 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:mime-version:subject:date:references
         :x-mailer;
        b=sI6p73txJgJvSVcBZQWF8Xr6vMj3i2nOeC7CbRw6PzGv2ZgKS50GciYfUavQek7ayW
         f8zbbIcJ/q+QES8zDOftyEeIqKMtUhGWSb+ib0mIX5kpZ6+8Dq1NCK160G/j+wNwWDU1
         E0Ev+YAp6gs9WSk/0CzoCdqmFbfEaQX1Uu/Vs=
Message-Id: <E9F6EC5D-B235-4503-838F-A3269725EE0D@gmail.com>
From: Rob Davies <rajdavies@gmail.com>
To: users@activemq.apache.org
In-Reply-To: <20091008143903.GA10920@towelie.gateway.2wire.net>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Issue creating a distributed queue using store and forward.
Date: Thu, 8 Oct 2009 16:22:11 +0100
References: <afa9caf10910071009w3605a890g6e1c091c435cf1cc@mail.gmail.com>
 <25791796.post@talk.nabble.com>
 <20091007190608.GA7334@towelie.corp.sodahead.com>
 <868BAA1C-6035-4A61-8373-9D66845A874D@gmail.com>
 <20091008143903.GA10920@towelie.gateway.2wire.net>

do you set the prefetchSize property on the network itself ? Setting  
it on the consumers wouldn't make any difference
On 8 Oct 2009, at 15:39, Eric Van Dewoestine wrote:

> I tried setting the prefetchSize to 1 for both consumers, but
> unfortunately, that didn't resolve the issue.  Messages still end up
> stuck on B2 when C2 is stopped.
>
> On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote:
>> set the prefetchSize=1 on your network - to avoid orphaned messages  
>> on
>> B2
>>
>> On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
>>
>>> Hmm, that seems like a flawed approach if the broker it visited has
>>> changed state and is now eligible to process those messages.  How is
>>> this type of fail over intended to be handled with activemq?  Is  
>>> there
>>> an alternate solution that I'm missing?
>>>
>>> On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
>>>>
>>>> I think that by design a message will not be forwarded to a broker
>>>> that it
>>>> has already visited.
>>>>
>>>> Joe
>>>> http://www.ttmsolutions.com
>>>>
>>>>
>>>> Eric Van wrote:
>>>>>
>>>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots  
>>>>> listing)
>>>>>
>>>>> I'm running into an issue with the store and forward feature of
>>>>> activemq, which I using in an attempt to create a highly available
>>>>> distributed queue.  I'm trying to figure out if the issue is a
>>>>> misconfiguration on my part, expected behavior of activemq, or a
>>>>> bug.
>>>>>
>>>>> The summary of the problem is that given 2 brokers, B1 and B2,  
>>>>> which
>>>>> each have one consumer, C1 and C2, which are subscribed to the  
>>>>> same
>>>>> queue.  If I stop a consumer on one of the brokers, the pending
>>>>> messages from that broker are not always forwarded to the other
>>>>> broker
>>>>> which still has a consumer, leading to those messages getting
>>>>> indefinitely stuck.
>>>>>
>>>>> The steps I use to reproduce this scenario are as follows (Note:  
>>>>> all
>>>>> producing and consuming is performed over the stomp transport):
>>>>>
>>>>> Couple notes about the consumers:
>>>>> - they have a prefetchSize of 40
>>>>> - the processing of messages can take some time, so for the  
>>>>> purposes
>>>>> of this exercise, I've created a simple consumer that sleeps for  
>>>>> 10
>>>>> seconds before sending the message ack (using client-individual  
>>>>> ack
>>>>> mode)
>>>>>
>>>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are  
>>>>> not
>>>>>  yet running.
>>>>> 2. produce a few thousand messages to B1
>>>>>  Note: B1 now has a few thousand pending messages and B2 has 0.
>>>>> 3. start consumer C2 (listing for messages from B2)
>>>>>  Note: messages are are successfully received and begin processing
>>>>>  (monitoring the brokers shows pending messages decreasing). Now  
>>>>> B2
>>>>>  has all the pending messages and B1 has 0.
>>>>> 4. start consumer C1 (listing for messages from B1)
>>>>>  Note: no messages are received, which is another issue I have
>>>>> since
>>>>>  B2 now has thousands of pending messages which C1 could help
>>>>>  process, but instead sits idle while C2 is forced to handle all
>>>>> the
>>>>>  messages.
>>>>> 5. stop consumer C2
>>>>>  Note: now I have thousands of messages sitting on B2 and 0 on B1
>>>>>  where a C1 is alive and ready to handle them.  So at this point,
>>>>>  despite having a consumer running, thousands of messages are  
>>>>> stuck
>>>>>  in the queue.
>>>>> 6. stop consumer C1
>>>>>  Note: now I have no consumers.  Stopping and restarting C1 has no
>>>>>  effect on the pending messages sitting on B1's queue.
>>>>> 7. stop both brokers
>>>>> 8. start B1, then start B2
>>>>> 9. start C1
>>>>>  Note: now all messages have migrated from B2 to B1 and C1 is  
>>>>> again
>>>>>  processing messages.
>>>>>
>>>>> So after step 5, the only way to recover from the stuck messages
>>>>> is to
>>>>> restart the brokers.
>>>>>
>>>>> Below is my current connector config which I have on both brokers.
>>>>> I've tried playing with the various properties of the connector,  
>>>>> but
>>>>> it seems as though no matter what I try the above scenario  
>>>>> continues
>>>>> to occur.
>>>>>
>>>>> <networkConnector
>>>>>   name="default-nc"
>>>>>   uri="multicast://default?group=${broker.group}"
>>>>>   dynamicOnly="true"
>>>>>   networkTTL="25"
>>>>>   suppressDuplicateQueueSubscriptions="true"/>
>>>>>
>>>>>
>>>>> So, is this an activemq bug? Am I mis-using activemq? Is there  
>>>>> some
>>>>> other way to achieve a highly available distributed queue?
>>>>>
>>>>> Any help in this regard is greatly appreciated.
>>>>>
>>>>> --
>>>>> eric
>>>>>
>>>>>
>>>
>>> --
>>> eric
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>
> --
> eric

Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/