activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fateev, Maxim" <fat...@amazon.com>
Subject RE: Paging support
Date Tue, 05 Sep 2006 19:04:28 GMT
Cool, it is really close to the design I was working on. I think there are a few issues with
it that should be addressed:

1. Queue should not keep list of references to all messages in flight. It leads to requirement
that MessageStore should deal with redriving messages that weren't acked. It also means that
messages should never be dispatched directly to consumers from producers as it happens right
now in Queue.send(...) method. They should always be put into MessageStore.
2. Such MessageStore API is not really compatible with message selectors in case of Queues.
For example if you have single consumer with message selector that selects one message from
1000 and you decide to keep 500 messages in memory then this consumer will not get its messages
ever. IMHO message selectors in Queues cannot be implemented really efficiently without keeping
all messages in memory and thus should not be supported.
3. It is not compatible with current subscription mechanism. Subscriptions should either stop
accepting messages when prefetch buffers are full or poll for messages when needed. But current
implementation expects that ALL messages are pushed to every subscription.
3. getNextMessage(...) should block if there are no messages in a queue. Otherwise it would
require busy polling.
4. I'm not sure about semantic of getNextMessage(..., RecoveryListener) API. Will it create
a thread per request? Will single blocked listener block the whole queue? I think "List getNextMessage(...,
int count)" would have cleaner semantic.

I think the proposed changes are too big and potentially dangerous to be put as fixes to existing
Queue and Topic implementations. I would recommend implementing alternative Queue and Topic
that use this new Store and keep existing ones until new one is stable. Recently introduced
"DestinationFactory" can be used to initialize alternative destination implementations.

-----Original Message-----
From: Rob Davies [mailto:rajdavies@gmail.com] 
Sent: Tuesday, September 05, 2006 11:28 AM
To: activemq-users@geronimo.apache.org
Subject: Re: Paging support

There will be some changes to the message store api for the following:

getNextMessage(String clientId, String subsName) - next message that hasn't been acknowledged
getNextMessage(String clientId, String subsName, MessageId from,
RecoveryListener) - allow batches of messages to be processed

etc.

In addition the pending LinkedList in Prefetch Subscription has been replaced with a PendingMessageCursor
interface - to allow different implementations to be plugged in.

So a durable subscriber will not have to be recovered - but can retrieve messages in batches
- taking into account the available memory in the message store.

There is a FilePendingMessageCursor - which can be used optionally for the temporary offline
storage for non-durable Topic subscribers


On 5 Sep 2006, at 18:35, Fateev, Maxim wrote:

> Rob,
> Would you give overview of the proposed design for the paging support 
> in ActiveMQ? The AMQ-845 doesn't provide any specifics.
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Tuesday, September 05, 2006 10:22 AM
> To: activemq-users@geronimo.apache.org
> Subject: Re: Paging support
>
> A message reference is kept in memory (which consists of a messageId 
> and information like sequence Id to enable faster retrieval from 
> storage).
> There is some work currently in progress in enable paging support in 
> ActiveMQ - see http://issues.apache.org/activemq/browse/AMQ-845
>
>
> On 5 Sep 2006, at 18:07, pradeep wrote:
>
>>
>> Maxim,
>>
>> Can you clarify this
>> Are you storing just the message id or the whole message in memory?
>> Your
>> help will be highly appreciated.
>>
>> Thanks
>> pradeep
>>
>>
>> Fateev, Maxim wrote:
>>>
>>> It would be interesting to see some numbers about memory utilization 
>>> per message in backlog.
>>>
>>> IMHO the main problem with keeping references in memory is that it 
>>> needs recovery phase on broker startup. It can take a while when 
>>> backlog is really big.
>>>
>>> -----Original Message-----
>>> From: James Strachan [mailto:james.strachan@gmail.com]
>>> Sent: Sunday, September 03, 2006 11:51 PM
>>> To: activemq-users@geronimo.apache.org
>>> Subject: Re: Paging support
>>>
>>> Note that MessageReferences != message contents; they are mostly 
>>> just a MessageId which is pretty small.
>>>
>>> On 8/29/06, Fateev, Maxim <fateev@amazon.com> wrote:
>>>> AFAIK messages are moved from journal to permantent storage
>>>> (JDBC) on
>>>> every checkpoint. The problem is that references for every message 
>>>> are still kept in memory.
>>>> IMHO it is inherent limitation of MessageStore API as it is defined 
>>>> now.
>>>> The only way to get message is by id. And the only way to have id 
>>>> is to keep it in memory.
>>>> I'm looking into changing MessageStore API to one that instead of
>>>>
>>>> Message         getMessage(MessageId identity)
>>>>
>>>> Would provide
>>>>
>>>> Message getNextMessage()
>>>>
>>>> method eliminating need to keep references in memory.
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Anders Bengtsson [mailto:ndrsbngtssn@yahoo.se]
>>>> Sent: Tuesday, August 29, 2006 1:23 AM
>>>> To: activemq-users@geronimo.apache.org
>>>> Subject: Re: Paging support
>>>>
>>>>
>>>>
>>>> gnodet wrote:
>>>>>
>>>>> If you don' t use any persistent store, messages must be kept in 
>>>>> memory, so you are obvisouly limited by the available mem.
>>>>>
>>>>
>>>> Yes, this is what I would expect.
>>>>
>>>>
>>>>
>>>>> If you use a jdbc store, messages can be removed from memory and 
>>>>> store for later consumption.
>>>>>
>>>>
>>>> Yes, except it doesn't actually work!
>>>>
>>>> It writes things to the store, but it is still very much limited by 
>>>> memory for some reason. (Someone reported this as AMQ-845 earlier, 
>>>> but with no response).
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Paging-support-tf2163517.html#a6034790
>>>> Sent from the ActiveMQ - User forum at Nabble.com.
>>>>
>>>
>>>
>>> --
>>>
>>> James
>>> -------
>>> http://radio.weblogs.com/0112098/
>>>
>>>
>>
>> --
>> View this message in context: http://www.nabble.com/Paging-support-
>> tf2163517.html#a6156162
>> Sent from the ActiveMQ - User forum at Nabble.com.
>>

Mime
View raw message