activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From taxingmonk <mking...@webhanger.com>
Subject Re: Message Group Overlaps and Load Balancing
Date Tue, 21 Nov 2006 16:17:19 GMT

Hi James,

To quote: "If you just want load balancing you could use 1 queue with a
selector
for each group." 

Yes, that'd be fine, the problem is that I want an overlap. Having done
further testing I realise this is a flawed notion in and of itself, I'm
inadvertently expecting a queue to duplicate messages for the overlapping
selectors.

Another way to describe the intending behaviour would be to have a set of
consumers on a topic where groups of consumers could be made where one
consumer in the group and only one receives a copy of the message for that
group. Thus I can build a replicated distributed queue. The net result is
actually identical to using multiple queues and composite destinations.

I think your suggestion:  "So send a message to an input queue then have a
farm of routers consuming them and routing them to the correct queue for the
right group to process it." is the closest, when I was first thinking up a
solution for this using composite destinations I was hoping to hide the
building of the composite destination from the sender.  This seems like a
nice way to do it.

Thanks James,
Max
p.s. Are you in "Norf Lundun" ATM?






James.Strachan wrote:
> 
> On 11/21/06, taxingmonk <mkington@webhanger.com> wrote:
>>
>> Hi Guys,
>>
>> I've got a use case where I want to load balance a given queue but
>> paritioned accross selectors.  I am convinced that somewhere between
>> composite destinations and message groups I can achieve what I'm after.
>>
>> The use case is that I want to be able to send one message which is load
>> balanced to a series of receievers which I group together somehow but at
>> the
>> same time I have multiple groups also receiving the same message.
>>
>> So In my example I have message M1 and I have Consumers (C1,C2,C3) Group
>> 1
>> and (C4,C5,C6) Group 2 and (C7,C8,C9) Group 3. I want message M1 to be
>> receieved by one member of Group 1, Group 2 and Group 3.
> 
> You could just use 3 queues for each group - then you can browse what
> messages are available for each group & monitor the queue depth etc.
> Thats the main benefit over using selectors.
> 
> Note that virtual destinations & composite destinations result in the
> message going to all of the destinations - in this use case you want
> it to only go to one of them. So selectors or a custom router is the
> best solution.
> 
> 
>> I thought that might be achieveable by using composite destinations but I
>> didn't want to manage a seperate queue for each group so that I could
>> drop a
>> message on the bus and have to arrive at all of them without specifying
>> send
>> (a.group1, a.group2, a.group3), etc, etc.
>>
>> Furthermore, what would be even better is if this could derived from the
>> message itself.  If we consider a numberline where I can specify a range
>> from say 1 to 10 and group 1 is interested in 1-3 and group 2 is
>> interested
>> 2-6 and group 3 is interested in 5-10. From there I would like to be able
>> to
>> say this message is for range 1
>>
>> It's almost like I want layer selector behaviour accross the top of the
>> message groups while load balancing :)
> 
> Just to be clear - Message Groups are basically all about sticky load
> balancing. All queues - whether using selectors or not - implement
> load balancing.
> 
> If you just want load balancing you could use 1 queue with a selector
> for each group. Using message groups on top of this should give you
> sticky load balancing if you need it.
> 
> 
>> Ultimatly this allows me to spawn
>> more consumers if I start to realise that the processes that are handling
>> a
>> particular range have a large data density, the key thing is that
>> multiple
>> groups can return responses to the one message, ranging specifying
>> partitions of data, which may or may not overlap. This I fear may be a
>> bridge too far.
>>
>> It should be noted that none of these messages is transactionally
>> important,
>> nor do I need the total guarentee that two or more consumers in the same
>> group receieve the same message, although that shouldn't be the common
>> case.
> 
> You could write a custom Broker interceptor to implement whatever
> routing policy you wish - using the message headers / body to route to
> a different queue based on some rules...
> http://incubator.apache.org/activemq/interceptors.html
> 
> Though its a bit more MOM-like to do these kinds of routing policies
> using a consumer which does the routing for you - which can then be as
> simple or complex as you wish. So send a message to an input queue
> then have a farm of routers consuming them and routing them to the
> correct queue for the right group to process it.
> 
> Am not sure if you need sticky load balancing from your mail - so
> maybe just 1 queue with selectors would do the trick?
> -- 
> 
> James
> -------
> http://radio.weblogs.com/0112098/
> 
> 

-- 
View this message in context: http://www.nabble.com/Message-Group-Overlaps-and-Load-Balancing-tf2678713.html#a7474172
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message