activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: Message Group Overlaps and Load Balancing
Date Tue, 21 Nov 2006 13:30:18 GMT
On 11/21/06, taxingmonk <mkington@webhanger.com> wrote:
>
> Hi Guys,
>
> I've got a use case where I want to load balance a given queue but
> paritioned accross selectors.  I am convinced that somewhere between
> composite destinations and message groups I can achieve what I'm after.
>
> The use case is that I want to be able to send one message which is load
> balanced to a series of receievers which I group together somehow but at the
> same time I have multiple groups also receiving the same message.
>
> So In my example I have message M1 and I have Consumers (C1,C2,C3) Group 1
> and (C4,C5,C6) Group 2 and (C7,C8,C9) Group 3. I want message M1 to be
> receieved by one member of Group 1, Group 2 and Group 3.

You could just use 3 queues for each group - then you can browse what
messages are available for each group & monitor the queue depth etc.
Thats the main benefit over using selectors.

Note that virtual destinations & composite destinations result in the
message going to all of the destinations - in this use case you want
it to only go to one of them. So selectors or a custom router is the
best solution.


> I thought that might be achieveable by using composite destinations but I
> didn't want to manage a seperate queue for each group so that I could drop a
> message on the bus and have to arrive at all of them without specifying send
> (a.group1, a.group2, a.group3), etc, etc.
>
> Furthermore, what would be even better is if this could derived from the
> message itself.  If we consider a numberline where I can specify a range
> from say 1 to 10 and group 1 is interested in 1-3 and group 2 is interested
> 2-6 and group 3 is interested in 5-10. From there I would like to be able to
> say this message is for range 1
>
> It's almost like I want layer selector behaviour accross the top of the
> message groups while load balancing :)

Just to be clear - Message Groups are basically all about sticky load
balancing. All queues - whether using selectors or not - implement
load balancing.

If you just want load balancing you could use 1 queue with a selector
for each group. Using message groups on top of this should give you
sticky load balancing if you need it.


> Ultimatly this allows me to spawn
> more consumers if I start to realise that the processes that are handling a
> particular range have a large data density, the key thing is that multiple
> groups can return responses to the one message, ranging specifying
> partitions of data, which may or may not overlap. This I fear may be a
> bridge too far.
>
> It should be noted that none of these messages is transactionally important,
> nor do I need the total guarentee that two or more consumers in the same
> group receieve the same message, although that shouldn't be the common case.

You could write a custom Broker interceptor to implement whatever
routing policy you wish - using the message headers / body to route to
a different queue based on some rules...
http://incubator.apache.org/activemq/interceptors.html

Though its a bit more MOM-like to do these kinds of routing policies
using a consumer which does the routing for you - which can then be as
simple or complex as you wish. So send a message to an input queue
then have a farm of routers consuming them and routing them to the
correct queue for the right group to process it.

Am not sure if you need sticky load balancing from your mail - so
maybe just 1 queue with selectors would do the trick?
-- 

James
-------
http://radio.weblogs.com/0112098/

Mime
View raw message