activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: Paging support
Date Tue, 05 Sep 2006 19:31:59 GMT

On 5 Sep 2006, at 20:04, Fateev, Maxim wrote:

> Cool, it is really close to the design I was working on. I think  
> there are a few issues with it that should be addressed:
>
> 1. Queue should not keep list of references to all messages in  
> flight. It leads to requirement that MessageStore should deal with  
> redriving messages that weren't acked. It also means that messages  
> should never be dispatched directly to consumers from producers as  
> it happens right now in Queue.send(...) method. They should always  
> be put into MessageStore.
Messages are always put into storage - but they are additionally  
pushed to subscribers for dispatching. I was going to implement a  
mixed approach - where consumers that can keep up with the producer  
will not be impeded by waiting for a poll, but slower/non-active  
consumers would ultimately rely on a poll from the store.
> 2. Such MessageStore API is not really compatible with message  
> selectors in case of Queues. For example if you have single  
> consumer with message selector that selects one message from 1000  
> and you decide to keep 500 messages in memory then this consumer  
> will not get its messages ever. IMHO message selectors in Queues  
> cannot be implemented really efficiently without keeping all  
> messages in memory and thus should not be supported.
Message selectors for Queues will always be an issue - but  
unfortunately the use of a message selector on a Queue is part of the  
JMS specification. We could make recommendations that if using  
selectors, we recommend that there should always be a set of active  
consumers able to select all messages from the Queue.
> 3. It is not compatible with current subscription mechanism.  
> Subscriptions should either stop accepting messages when prefetch  
> buffers are full or poll for messages when needed. But current  
> implementation expects that ALL messages are pushed to every  
> subscription.
Yes - am changing - so the PendingMessageCursor can in effect ignore  
new messages from a producer if it is full - or the usage manager has  
reached it's limit - and poll for them later.
> 3. getNextMessage(...) should block if there are no messages in a  
> queue. Otherwise it would require busy polling.
depends on the implementation - more messages available can be  
triggered from producer etc - providing the PendingMessageCursor is  
intialized with state and maintains it.
> 4. I'm not sure about semantic of getNextMessage(...,  
> RecoveryListener) API. Will it create a thread per request? Will  
> single blocked listener block the whole queue? I think "List  
> getNextMessage(..., int count)" would have cleaner semantic.
I think a Listener allows for more flexibility - in allowing the  
PendingMessageCursor to stop a request if the usage memory is at its  
limit for example
>
> I think the proposed changes are too big and potentially dangerous  
> to be put as fixes to existing Queue and Topic implementations. I  
> would recommend implementing alternative Queue and Topic that use  
> this new Store and keep existing ones until new one is stable.  
> Recently introduced "DestinationFactory" can be used to initialize  
> alternative destination implementations.

Difficult call - I'm trying not to change the existing code too much  
- as I know there's been alot of bug fixes in the  
PrefetchSubscription/DurableTopicSubscriber etc.  code, for timing  
issues, transactions etc - I'm tempted not to loose that if possible :)

>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Tuesday, September 05, 2006 11:28 AM
> To: activemq-users@geronimo.apache.org
> Subject: Re: Paging support
>
> There will be some changes to the message store api for the following:
>
> getNextMessage(String clientId, String subsName) - next message  
> that hasn't been acknowledged getNextMessage(String clientId,  
> String subsName, MessageId from,
> RecoveryListener) - allow batches of messages to be processed
>
> etc.
>
> In addition the pending LinkedList in Prefetch Subscription has  
> been replaced with a PendingMessageCursor interface - to allow  
> different implementations to be plugged in.
>
> So a durable subscriber will not have to be recovered - but can  
> retrieve messages in batches - taking into account the available  
> memory in the message store.
>
> There is a FilePendingMessageCursor - which can be used optionally  
> for the temporary offline storage for non-durable Topic subscribers
>
>
> On 5 Sep 2006, at 18:35, Fateev, Maxim wrote:
>
>> Rob,
>> Would you give overview of the proposed design for the paging support
>> in ActiveMQ? The AMQ-845 doesn't provide any specifics.
>>
>> -----Original Message-----
>> From: Rob Davies [mailto:rajdavies@gmail.com]
>> Sent: Tuesday, September 05, 2006 10:22 AM
>> To: activemq-users@geronimo.apache.org
>> Subject: Re: Paging support
>>
>> A message reference is kept in memory (which consists of a messageId
>> and information like sequence Id to enable faster retrieval from
>> storage).
>> There is some work currently in progress in enable paging support in
>> ActiveMQ - see http://issues.apache.org/activemq/browse/AMQ-845
>>
>>
>> On 5 Sep 2006, at 18:07, pradeep wrote:
>>
>>>
>>> Maxim,
>>>
>>> Can you clarify this
>>> Are you storing just the message id or the whole message in memory?
>>> Your
>>> help will be highly appreciated.
>>>
>>> Thanks
>>> pradeep
>>>
>>>
>>> Fateev, Maxim wrote:
>>>>
>>>> It would be interesting to see some numbers about memory  
>>>> utilization
>>>> per message in backlog.
>>>>
>>>> IMHO the main problem with keeping references in memory is that it
>>>> needs recovery phase on broker startup. It can take a while when
>>>> backlog is really big.
>>>>
>>>> -----Original Message-----
>>>> From: James Strachan [mailto:james.strachan@gmail.com]
>>>> Sent: Sunday, September 03, 2006 11:51 PM
>>>> To: activemq-users@geronimo.apache.org
>>>> Subject: Re: Paging support
>>>>
>>>> Note that MessageReferences != message contents; they are mostly
>>>> just a MessageId which is pretty small.
>>>>
>>>> On 8/29/06, Fateev, Maxim <fateev@amazon.com> wrote:
>>>>> AFAIK messages are moved from journal to permantent storage
>>>>> (JDBC) on
>>>>> every checkpoint. The problem is that references for every message
>>>>> are still kept in memory.
>>>>> IMHO it is inherent limitation of MessageStore API as it is  
>>>>> defined
>>>>> now.
>>>>> The only way to get message is by id. And the only way to have id
>>>>> is to keep it in memory.
>>>>> I'm looking into changing MessageStore API to one that instead of
>>>>>
>>>>> Message         getMessage(MessageId identity)
>>>>>
>>>>> Would provide
>>>>>
>>>>> Message getNextMessage()
>>>>>
>>>>> method eliminating need to keep references in memory.
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Anders Bengtsson [mailto:ndrsbngtssn@yahoo.se]
>>>>> Sent: Tuesday, August 29, 2006 1:23 AM
>>>>> To: activemq-users@geronimo.apache.org
>>>>> Subject: Re: Paging support
>>>>>
>>>>>
>>>>>
>>>>> gnodet wrote:
>>>>>>
>>>>>> If you don' t use any persistent store, messages must be kept in
>>>>>> memory, so you are obvisouly limited by the available mem.
>>>>>>
>>>>>
>>>>> Yes, this is what I would expect.
>>>>>
>>>>>
>>>>>
>>>>>> If you use a jdbc store, messages can be removed from memory and
>>>>>> store for later consumption.
>>>>>>
>>>>>
>>>>> Yes, except it doesn't actually work!
>>>>>
>>>>> It writes things to the store, but it is still very much  
>>>>> limited by
>>>>> memory for some reason. (Someone reported this as AMQ-845 earlier,
>>>>> but with no response).
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Paging-support-tf2163517.html#a6034790
>>>>> Sent from the ActiveMQ - User forum at Nabble.com.
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> James
>>>> -------
>>>> http://radio.weblogs.com/0112098/
>>>>
>>>>
>>>
>>> --
>>> View this message in context: http://www.nabble.com/Paging-support-
>>> tf2163517.html#a6156162
>>> Sent from the ActiveMQ - User forum at Nabble.com.
>>>


Mime
View raw message