Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: neutral (athena.apache.org: 196.38.97.11 is neither permitted
 nor denied by domain of roelof.naude@gmail.com)
Message-ID: <4CD0176A.9070804@gmail.com>
Date: Tue, 02 Nov 2010 15:51:38 +0200
From: Roelof Naude <roelof.naude@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.1.12) Gecko/20100907 Fedora/3.0.7-1.fc12 Lightning/1.0b2pre
 Thunderbird/3.0.7
MIME-Version: 1.0
To: users@activemq.apache.org
Subject: Re: Virtual Topic and missed messages gone missing with
 selectorAware=true
References: <9C408626-C3E2-4346-9CD4-2CF4C20D8C1D@homeaway.co.uk>
 <AANLkTinAL-GKYtQWhq=ukJbxy+vaq6_4nb25JXwHQ5A2@mail.gmail.com>
 <EEF833A5-6FD9-4FA7-89DC-9FC5CE093F13@homeaway.co.uk>
 <6674E6AE-A80D-47BD-B40B-73CF38A4594C@homeaway.co.uk>
 <9f2dd109478de492d85b3bc201cd3690@mail.svenvintges.nl>
 <8A013711-2613-450A-A487-379E784AF1D6@homeaway.co.uk>
In-Reply-To: <8A013711-2613-450A-A487-379E784AF1D6@homeaway.co.uk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

hi maciej,

that sounds interesting. would you mind sharing the filter and changes 
to SelectorAwareVirtualTopicInterceptor?

On 11/02/2010 03:48 PM, Maciej Rakowicz wrote:
> Ok, i do have a somewhat contrived workaround. Basically, I decided to
> cache selector string of any queue subscription (well vtopic queue)
> that I intercept (BrokerFilter). Then I have modified
> org
> .apache
> .activemq.broker.region.virtual.SelectorAwareVirtualTopicInterceptor
> so that if there is no subscription for a destination (here we talk
> about queue rather than vtopic again) it tries looking it up in the
> cache - well, just the selector string as it's all I need, plus due to
> the way cache is distributed it has to be easily serializable. It
> compiles the selector and tries to match it against
> MessageEvaluationContext. Of course this works with several
> assumptions in mind. First, I need this consumer to be connected at
> least once (that's when I can cache its selector). Secondly, selector
> <->  queue is 1-to-1 relation. That's perfectly acceptable in my setup.
>
> m.
>
> On 1 Nov 2010, at 12:16, Sven Vintges wrote:
>
>> Perhaps a work-around would be to create a garbage-service. Ie. a
>> services that pops off all messages not needed by the consumer to /
>> dev/null? Though you probably don't know which messages are not
>> needed by the subscriber since you don't know which subscribers you
>> will have... Perhaps you could create an algorithm that does
>> something like "if the consumer is up and the message is older than
>> xxx time" pop it off?
>>
>> On Mon, 1 Nov 2010 10:25:40 +0000, "Maciej Rakowicz"<maciek@homeaway.co.uk
>>> wrote:
>>> in case of selectorAware=true and consumer down, what happens to
>>> those  messages? are they discarded? Is there anything sent to one of
>>> the  advisories? I can't seem to find it anywhere.
>>>
>>> On 1 Nov 2010, at 09:48, Maciej Rakowicz wrote:
>>>
>>>> I guess, only if we keep downtimes short enough so 'matched'
>>>> messages don't expire and big enough maxPageSize or something to
>>>> keep matched messages consumed.
>>>>
>>>> Does anyone know if this (i.e. AMQ-3004) is going to be worked on
>>>> in the near future?
>>>>
>>>> m.
>>>>
>>>> On 1 Nov 2010, at 09:32, Gary Tully wrote:
>>>>
>>>>> Message expiry may help in the short term, set a time to live on
>>>>> the
>>>>> messages such that the built up unmatched messages expire and are
>>>>> removed.
>>>>>
>>>>> On 1 November 2010 09:20, Maciej Rakowicz
>>>>> <maciek@homeaway.co.uk>   wrote:
>>>>>> All,
>>>>>>
>>>>>> amq 5.4.0. jdk 1.6
>>>>>>
>>>>>> Scenario:
>>>>>> broker persistence=true, selectorAware=false
>>>>>> Virtual Topic with say one queue. There is a consumer A
>>>>>> connected  to that
>>>>>> queue with a selector. All works fine. Consumer A dies,
>>>>>> messages  pile up,
>>>>>> consumer A starts back up, missed messages are redelivered. You
>>>>>> can easily
>>>>>> verify that queue receives all posted messages while consumer's
>>>>>> down.
>>>>>> Now, adding consumer B, since selectorAware=false and consumer
>>>>>> B  uses an
>>>>>> exclusive selector messages sent to consumer A are not consumed
>>>>>> by consumer
>>>>>> B. All good save the fact that all unmatched messages end up
>>>>>> polluting
>>>>>> consumer's B queue. All according to the documentation.
>>>>>> Flipping selectorAware to true solves one problem but
>>>>>> introduces  another
>>>>>> (way more important in my setup). Unmatched messages won't pile
>>>>>> up on
>>>>>> consumer's B queue which is fine. However, if consumer A dies
>>>>>> they are not
>>>>>> sent to consumer's A queue (disappearing after reaching the
>>>>>> topic  - I dunno
>>>>>> where they go) hence disabling any missed message delivery on
>>>>>> consumer's A
>>>>>> startup.
>>>>>>
>>>>>> I there any way around it? I really need to handle all missed
>>>>>> messages with
>>>>>> no exception. At the same time I cannot allow stacking up
>>>>>> messages in each
>>>>>> and every queue although they are not matched.
>>>>>>
>>>>>> Btw. I do not have a list of consumers in advance, the
>>>>>> subscription is fully
>>>>>> dynamic, although since broker runs persistence, once subscribed
>>>>>> I do have
>>>>>> an idea of who's subscribed even if connection is currently down.
>>>>>>
>>>>>> I would appreciate any hints.
>>>>>>
>>>>>> mac
>>>>>
>>>>>
>>>>>
>>>>> -- http://blog.garytully.com
>>>>> http://fusesource.com
>>>>
>>
>