activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: Message Group Overlaps and Load Balancing
Date Tue, 21 Nov 2006 16:38:36 GMT
So if you want overlap - where 1 message may go to several groups, but
each group processes it once - then multiple queues is the best
solution I think. So either use a consumer to do the routing, or a
broker interceptor.

BTW yeah am still based in Islington :)

On 11/21/06, taxingmonk <mkington@webhanger.com> wrote:
>
> Hi James,
>
> To quote: "If you just want load balancing you could use 1 queue with a
> selector
> for each group."
>
> Yes, that'd be fine, the problem is that I want an overlap. Having done
> further testing I realise this is a flawed notion in and of itself, I'm
> inadvertently expecting a queue to duplicate messages for the overlapping
> selectors.
>
> Another way to describe the intending behaviour would be to have a set of
> consumers on a topic where groups of consumers could be made where one
> consumer in the group and only one receives a copy of the message for that
> group. Thus I can build a replicated distributed queue. The net result is
> actually identical to using multiple queues and composite destinations.
>
> I think your suggestion:  "So send a message to an input queue then have a
> farm of routers consuming them and routing them to the correct queue for the
> right group to process it." is the closest, when I was first thinking up a
> solution for this using composite destinations I was hoping to hide the
> building of the composite destination from the sender.  This seems like a
> nice way to do it.
>
> Thanks James,
> Max
> p.s. Are you in "Norf Lundun" ATM?
>
>
>
>
>
>
> James.Strachan wrote:
> >
> > On 11/21/06, taxingmonk <mkington@webhanger.com> wrote:
> >>
> >> Hi Guys,
> >>
> >> I've got a use case where I want to load balance a given queue but
> >> paritioned accross selectors.  I am convinced that somewhere between
> >> composite destinations and message groups I can achieve what I'm after.
> >>
> >> The use case is that I want to be able to send one message which is load
> >> balanced to a series of receievers which I group together somehow but at
> >> the
> >> same time I have multiple groups also receiving the same message.
> >>
> >> So In my example I have message M1 and I have Consumers (C1,C2,C3) Group
> >> 1
> >> and (C4,C5,C6) Group 2 and (C7,C8,C9) Group 3. I want message M1 to be
> >> receieved by one member of Group 1, Group 2 and Group 3.
> >
> > You could just use 3 queues for each group - then you can browse what
> > messages are available for each group & monitor the queue depth etc.
> > Thats the main benefit over using selectors.
> >
> > Note that virtual destinations & composite destinations result in the
> > message going to all of the destinations - in this use case you want
> > it to only go to one of them. So selectors or a custom router is the
> > best solution.
> >
> >
> >> I thought that might be achieveable by using composite destinations but I
> >> didn't want to manage a seperate queue for each group so that I could
> >> drop a
> >> message on the bus and have to arrive at all of them without specifying
> >> send
> >> (a.group1, a.group2, a.group3), etc, etc.
> >>
> >> Furthermore, what would be even better is if this could derived from the
> >> message itself.  If we consider a numberline where I can specify a range
> >> from say 1 to 10 and group 1 is interested in 1-3 and group 2 is
> >> interested
> >> 2-6 and group 3 is interested in 5-10. From there I would like to be able
> >> to
> >> say this message is for range 1
> >>
> >> It's almost like I want layer selector behaviour accross the top of the
> >> message groups while load balancing :)
> >
> > Just to be clear - Message Groups are basically all about sticky load
> > balancing. All queues - whether using selectors or not - implement
> > load balancing.
> >
> > If you just want load balancing you could use 1 queue with a selector
> > for each group. Using message groups on top of this should give you
> > sticky load balancing if you need it.
> >
> >
> >> Ultimatly this allows me to spawn
> >> more consumers if I start to realise that the processes that are handling
> >> a
> >> particular range have a large data density, the key thing is that
> >> multiple
> >> groups can return responses to the one message, ranging specifying
> >> partitions of data, which may or may not overlap. This I fear may be a
> >> bridge too far.
> >>
> >> It should be noted that none of these messages is transactionally
> >> important,
> >> nor do I need the total guarentee that two or more consumers in the same
> >> group receieve the same message, although that shouldn't be the common
> >> case.
> >
> > You could write a custom Broker interceptor to implement whatever
> > routing policy you wish - using the message headers / body to route to
> > a different queue based on some rules...
> > http://incubator.apache.org/activemq/interceptors.html
> >
> > Though its a bit more MOM-like to do these kinds of routing policies
> > using a consumer which does the routing for you - which can then be as
> > simple or complex as you wish. So send a message to an input queue
> > then have a farm of routers consuming them and routing them to the
> > correct queue for the right group to process it.
> >
> > Am not sure if you need sticky load balancing from your mail - so
> > maybe just 1 queue with selectors would do the trick?
> > --
> >
> > James
> > -------
> > http://radio.weblogs.com/0112098/
> >
> >
>
> --
> View this message in context: http://www.nabble.com/Message-Group-Overlaps-and-Load-Balancing-tf2678713.html#a7474172
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 

James
-------
http://radio.weblogs.com/0112098/

Mime
View raw message