activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: Mitigating uneven queue consumption
Date Tue, 20 Jun 2006 06:41:14 GMT
BTW you are using version 4.0 right?

The forced-reconnect after a certain amount of time is probably the
simplest way of solving this issue.

Another option could just be to load balance the clients on a broker
by periodically connecting to all available brokers using JMX, seeing
what the distribution of clients are and if there are too many on a
certain broker, then disconnect them and force a reconnect.

I'm also interested in the multiplexing option. We already have the
FanoutTransport which kinda does something like this; though we'd need
to think carefully about the semantics, like transactions etc.

On 6/19/06, John Heitmann <jheitmann@gmail.com> wrote:
> Our setup: We have a set of brokers directly feeding consumers
> messages on a single queue destination (I'm going to lie a bit about
> our setup to keep things simple). On each host we're running a broker
> and multiple consumers. Each consumer connects to the broker set with
> a failover connection. Messages flow into the broker set from
> upstream sources (in our case another set of brokers) in a pretty
> even fashion, so each broker gets about the same amount of messages.
>
> The problem with this is that it's easy for one broker to have an
> uneven number of consumers, and in some cases have so few consumers
> that it will backlog. One way this can happen is if a broker is taken
> down for some routine reason. All the connections to that broker will
> fail over to another broker. When the broker comes back up it will
> not receive new connections until consumers restart yet it will
> continue to receive an equal share of messages from upstream.
>
> We're considering implementing two different code changes to ActiveMQ
> to help with this, but I wanted to poll the group to be sure there
> was not already something there to help us.
>
> Our first solution would be to add logic to ActiveMQConnection that
> would trigger a reconnect on the failover transport after a certain
> configurable time period expires and the current transaction is
> complete. This would give decent statistical certainty that a single
> broker won't ever get a large backlog compared to the others and
> should be transparent to the client code except for a hitch in
> message flow.
>
> Our second solution is to implement connection multiplexing for
> consumers. Behind the scenes each consumer would connect to all of
> the broker set, divide its prefetch buffer across all those brokers,
> and start grabbing messages from multiple at once. We'd also need
> hooks to auto-discovery for when we bring new brokers up or old
> brokers come back. Also transactions would probably need to be tied
> to a single broker. This seems like it would be smoother behavior
> overall, but tougher to implement than the first so we're putting it
> off. Scalability is also a concern.
>
> If you know of a way to do this already I would appreciate hearing
> about it. Thanks,
>
> John
>


-- 

James
-------
http://radio.weblogs.com/0112098/

Mime
View raw message