activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: Mitigating uneven queue consumption
Date Tue, 22 Aug 2006 12:23:09 GMT
On 8/22/06, John Heitmann <jheitmann@gmail.com> wrote:
> (moved to activemq-dev)
>
> On Jun 19, 2006, at 11:41 PM, James Strachan wrote:
>
> > The forced-reconnect after a certain amount of time is probably the
> > simplest way of solving this issue.
>
> I ended up implementing this first. It works ok, but it's pretty lame
> by design. It has actually turned out to be most handy to stress test
> the failover transport. Would you be interested in the patch for this
> or would you rather wait for multiplexing?

Sure :)


> > I'm also interested in the multiplexing option. We already have the
> > FanoutTransport which kinda does something like this; though we'd need
> > to think carefully about the semantics, like transactions etc.
>
> I am part way through the implementation of this and wanted to bounce
> the rough design off you and see if it sounds like the right
> direction.

Great! :)

> I'm calling it 'fanin', but I see you've called something
> similar 'jedi' in Jira. Maybe I'll rename mine 'sith' just to be
> contrarian :)

LOL!


> It currently connects to all discovered brokers. Message sends round-
> robin to individual brokers (ie only one broker handles a message
> send).

BTW for topics you might wanna send the message to all brokers.


>  Connection/session metadata commands are handled with the same
> state tracking failover uses. Anything stored in the state tracker is
> broadcast instead of unicast. Message acks are obviously only sent to
> the broker they originated from. Ranged acks are intelligently split
> so each broker gets the  acks for the bits of range only it knows about.

Sounds *awesome*! Can't wait for the patch :)


> Transactions are still todo. My plan is that once the transport sees
> a transaction it punts and stops round-robin sending and locks down
> to one broker. That should be good enough for 1.0.

Yeah, that'd be fine to start with.

Another option is that each transaction is pinned to a specific broker
& all operations are fixed with that broker. Bear in mind different
threads can be performing different transactions - so if a Message /
MessageAck has a transactionId property then associate it with a
single broker. We could then round robin transactions around brokers
too :)


> Request/reply and
> temporary destinations are still untested, but I think they should
> work without much design tweaking.

Yeah - we'd need to broadcast temporary destination commands to all
brokers just in case (as the temporary destination could be put on the
JMSReplyTo on another message


> I don't plan on doing anything about message ordering. We don't care
> much about ordering, and I don't think there's much that could be
> done here anyway.

Cool. If forks are worried about that we could pin a destination to a
broker - or even a message for a destination with a specific Message
Group header (JMSXGroupID) to a broker so we could load balance
messages to the same queue across brokers using Message Groups to
preserve order


> Connection handling is done roughly the same way fanout does. The
> transport is basically a frankenstein of fanout and failover. Once I
> have the functionality complete I think it'll worthwhile to refactor
> the 3 a bit.
>
> Does this sounds pretty reasonable?

Sounds great. I'd be happy for this to replace fanout as its mostly a
case of just choosing the 1 or all brokers that messages go to based
on the type of command & whether or not its in a transaction etc.
Given the different semantics to failover its probably worth keeping
it separate though we could maybe reuse some code at some point.

Its all sounding great so far John! :)

-- 

James
-------
http://radio.weblogs.com/0112098/

Mime
View raw message