activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: Paging support
Date Wed, 06 Sep 2006 05:43:19 GMT

On 5 Sep 2006, at 21:18, Fateev, Maxim wrote:

>> Messages are always put into storage - but they are additionally  
>> pushed to subscribers for dispatching. I was going to implement a  
>> mixed approach - where consumers that can keep up with the  
>> producer will not be impeded by waiting for a poll, but slower/non- 
>> active consumers would ultimately rely on a poll from the store.
> My view is that this "pushing" functionality should be hidden  
> behind MessageStore API. CachingMessageStore could be created that  
> would keep messages for fast consumers in memory delegating other  
> requests to another permanent MessageStore.
This already happens  - see the Journal Persistence Adaptor.
> Having two separate dispatching code paths complicates code.
yes it does - but it's done for performance - polling is slow.

> Another problem with direct dispatch is that in case of synchronous  
> dispatch producer performance directly affected by performance of  
> consumer.
The performance of the consumer is currently taken into account  
whilst dispatching - in the Subscription classes - and the reality is  
that the producer doesn't always do the dispatching, the thread  
associated with the consumer does too - if it is slow.

> I don't think that it is really a good idea to have receiving  
> socket thread perform write to a consumer socket directly.
This is done for performance - there are defensive mechanisms built  
in to the transport layers to prevent blocked clients locking the  
broker - but at the end of the day this is optional - the consumer  
can decide if messages are dispatched asynchronously from within the  
broker.
>
>> Yes - am changing - so the PendingMessageCursor can in effect  
>> ignore new messages from a producer if it is full - or the usage  
>> manager has reached it's limit - and poll for them later.
> It is an example of two code paths. Only polling part would be  
> required if you eliminate pushing part.
But the two code paths happens currently in PrefetchSubscription
>
>>> 3. getNextMessage(...) should block if there are no messages in a
>>> queue. Otherwise it would require busy polling.
>> depends on the implementation - more messages available can be  
>> triggered from producer etc - providing the PendingMessageCursor  
>> is intialized with state and maintains it.
> Again blocking getNextMessage() would be much more simple if  
> pushing part is eliminated.
I think you've made the assumption that polling is best ?
>
>>> 4. I'm not sure about semantic of getNextMessage(...,
>>> RecoveryListener) API. Will it create a thread per request? Will
>>> single blocked listener block the whole queue? I think "List
>>> getNextMessage(..., int count)" would have cleaner semantic.
>> I think a Listener allows for more flexibility - in allowing the  
>> PendingMessageCursor to stop a request if the usage memory is at  
>> its limit for example
> I'm not sure why it has more flexibility. In case of blocking  
> get... call you have the same flexibility but more simple  
> threading. For example you just don't call getNextMessage if the  
> usage memory is at its limit. RecoveryListener would require  
> additional thread for every consumer which doesn't scale.
I don't see why additional threads are required ?
> BTW do you plan to have RecoveryListener per queue consumer or  
> single one for the whole queue?
>
>> Difficult call - I'm trying not to change the existing code too much
>> - as I know there's been alot of bug fixes in the  
>> PrefetchSubscription/DurableTopicSubscriber etc.  code, for timing  
>> issues, transactions etc - I'm >>tempted not to loose that if  
>> possible :)
> You can create different destination implementation that still uses  
> existing Subscription framework but relies on new MessageStore API.  
> I have an example of such code on my desktop.

The comment at the bottom:)

We have tried an implementation of the message store using polling  
before but the reason we prefer a hybrid approach (which is what we  
have in essence in ActiveMQ at the moment) - is performance - we get  
dramatically better performance for push based messaging (the general  
case) and pulling messages for slower consumers. I think the main  
difference between the implementation your working on and the current  
architecture is that currently we can optionally decide to push from  
the producer socket directly to consumers  - which is about as fast  
as it gets for broker message delivery - and we've taken on the  
additional complexity to keep this as an option.

But given the flexibility behind the architecture of ActiveMQ - there  
is nothing to stop having more than one Destination implementation -  
now that we've incorporated your patch for the DestinationFactory.

cheers,

Rob



Mime
View raw message