Mailing-List: contact activemq-users-help@geronimo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: activemq-users@geronimo.apache.org
Received-SPF: pass (asf.osuosl.org: domain of james.strachan@gmail.com
 designates 72.14.214.198 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=d3u9Vhc2AJnpdOrQ4OgrPRgFn8o4o6ZDjCSS6rKhrYG6Ifgq0tA3cuSQHA0EOArUSi0bDr6oesIHW02ivyL9hzYOns7Ga965HBy8kLT5/VaaRqc4H5IAntV2AOVxBMt9SlYXKh0qP5Ta9TF2T1LANXqx5ZClGJEs9QJCZr6b1o8=
Message-ID: <ec6e67fd0606192341n27cffae8tf6ffe388bf6c533c@mail.gmail.com>
Date: Tue, 20 Jun 2006 07:41:14 +0100
From: "James Strachan" <james.strachan@gmail.com>
To: activemq-users@geronimo.apache.org
Subject: Re: Mitigating uneven queue consumption
In-Reply-To: <82AB608D-F62C-49B4-94A2-8AC854A14505@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <82AB608D-F62C-49B4-94A2-8AC854A14505@gmail.com>

BTW you are using version 4.0 right?

The forced-reconnect after a certain amount of time is probably the
simplest way of solving this issue.

Another option could just be to load balance the clients on a broker
by periodically connecting to all available brokers using JMX, seeing
what the distribution of clients are and if there are too many on a
certain broker, then disconnect them and force a reconnect.

I'm also interested in the multiplexing option. We already have the
FanoutTransport which kinda does something like this; though we'd need
to think carefully about the semantics, like transactions etc.

On 6/19/06, John Heitmann <jheitmann@gmail.com> wrote:
> Our setup: We have a set of brokers directly feeding consumers
> messages on a single queue destination (I'm going to lie a bit about
> our setup to keep things simple). On each host we're running a broker
> and multiple consumers. Each consumer connects to the broker set with
> a failover connection. Messages flow into the broker set from
> upstream sources (in our case another set of brokers) in a pretty
> even fashion, so each broker gets about the same amount of messages.
>
> The problem with this is that it's easy for one broker to have an
> uneven number of consumers, and in some cases have so few consumers
> that it will backlog. One way this can happen is if a broker is taken
> down for some routine reason. All the connections to that broker will
> fail over to another broker. When the broker comes back up it will
> not receive new connections until consumers restart yet it will
> continue to receive an equal share of messages from upstream.
>
> We're considering implementing two different code changes to ActiveMQ
> to help with this, but I wanted to poll the group to be sure there
> was not already something there to help us.
>
> Our first solution would be to add logic to ActiveMQConnection that
> would trigger a reconnect on the failover transport after a certain
> configurable time period expires and the current transaction is
> complete. This would give decent statistical certainty that a single
> broker won't ever get a large backlog compared to the others and
> should be transparent to the client code except for a hitch in
> message flow.
>
> Our second solution is to implement connection multiplexing for
> consumers. Behind the scenes each consumer would connect to all of
> the broker set, divide its prefetch buffer across all those brokers,
> and start grabbing messages from multiple at once. We'd also need
> hooks to auto-discovery for when we bring new brokers up or old
> brokers come back. Also transactions would probably need to be tied
> to a single broker. This seems like it would be smoother behavior
> overall, but tougher to implement than the first so we're putting it
> off. Scalability is also a concern.
>
> If you know of a way to do this already I would appreciate hearing
> about it. Thanks,
>
> John
>


-- 

James
-------
http://radio.weblogs.com/0112098/