activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Terry K <tyrone.k...@thomson.com>
Subject Re: transacted sessions in multi-threaded client
Date Thu, 11 Oct 2007 23:27:08 GMT

Thank you for your comments Joe, 

both senerios you described result in a consumer/session being flooded with
messages up to the prefetch limit. I guess the  transacted multi-threaded
client code would need to evenly iterate the sessions looking for  whatever
sessions have  the messages. Setting the prefetch limit low shoud ensure a
large number of messages are distributed across the sessions while a low
number of messages can reasonabily route to a single session where all
messages get processed in a single thread.

I am pooling my  consumer/sessions and using each one to receive a message
and use it within a thread. Problem is, Apache Commons Pooling does not
allow iteration of items in a pool - to find the session with all the
messaages.

What is the lowest prefetch limit that would ensure good throughput of
messages without pulling too often from the broker

terry k



ttmdev wrote:
> 
> Here's what I have experienced wrt prefetch limits; you may find it
> useful. 
> 
> If a destination (queue) has active consumers, then as messages arrive at
> the destination, the broker will evenly dispatch the messages across the
> active consumers; no matter what the prefetch limit may be. 
> 
> Now let's say that you have a destination that has a bunch of pending
> messages, but no active consumers to send them to.   If you then activate
> say 5 consumers for that destination, the broker will latch on to the
> first of those 5 that connects to it and will stream the pending messages
> - up to the prefetch limit - to that first consumer. As soon as it
> dis-engages from the first one, it will then begin to 'evenly' dispatch
> messages to all 5 (i.e., assuming the other 4 have already connected w/the
> broker). If the prefetch limit is set to a value that is higher than or
> equal to the number of pending messages, then that first consumer will get
> ALL the messages.    
> 
> Since your message processing time is relatively small, you shouldn't have
> to have a prefetch limit of '1'.   
> 
> Hope this helps,   
> Joe
> 
> 
>  
> 
> Terry K wrote:
>> 
>> In a point-to-point queue I want to use transacted sessions to ensure
>> messages are processed completely before being removed from the queue.
>> The client is to be multi threaded to process messages in parallel.
>> 
>> I decided each thread would need it's own Session object (for
>> transacting) so that when the work on each message is successfully
>> completed the session can be commited. Session objects would be pooled
>> and retrieved when the next message is ready to be received.
>> 
>> The documentation states that, "ActiveMQ is a high performance message
>> bus that streams messages to consumers as fast as possible so that the
>> consumer always has a buffer of messages, in RAM, ready to process -
>> rather than have them explicitly pull messages from the server which adds
>> significant latency per message".
>> 
>> The problem is that messages will not be evenly distributed amoungst
>> Sessions, with the likeihood that a single session will be flooded with
>> all the messages making it impossible to distribute the work amoungst
>> multiple threads.
>> 
>> The logical next step would be to set the prefetch limit to one and have
>> a pull-one-message-at-a-time dispatch policy  (as described
>> here:http://activemq.apache.org/what-is-the-prefetch-limit-for.html).
>> Apparently, this is advised against as it's very SLOW
>> (http://osdir.com/ml/java.activemq.user/2005-03/msg00188.html). I expect
>> millions of messages in my queue, each one can be processed in a matter
>> of seconds.
>> 
>> The problem of multi threading and transacted sessions must be fairly
>> common - can anyone suggest other options?
>> 
>> Terry
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/transacted-sessions-in-multi-threaded-client-tf4603908s2354.html#a13166649
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message