activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Lloyd <jll...@silvertailsystems.com>
Subject Re: Single consumer for multiple brokers?
Date Thu, 29 Jan 2009 16:50:49 GMT
I'm interested in a variation of this for load balancing. Assume the volume
of data coming from publishers is very large, and you need N brokers (think
N is 10 or more). Furthermore, you want N+1 or N+2 redundancy so that if any
1 or 2 brokers die, there is sufficient capacity to continue handling all of
the traffic (after affected publishers reconnect). The brokers might all be
behind a hardware load balancer (e.g. F5 or NetScaler) so that all
publishers connect via a virtual IP.

Now, from the consumer side, we also have many consumers. Each consumer only
subscribes to a subset of the available topics, and we arrange via design of
our topics such that one consumer can always keep up with the volume of data
published on one topic. But the consumer must connect to every broker.

So, we need a fan-in variation of a failover transport that connects to
every broker and actively consumes data from all brokers. If any broker
disconnects, the consumer would use the exponential backoff reconnect logic
to reconnect when the broker becomes available.

So, the backup=true option isn't helpful. Instead, we want something like
"fan_in=true".

It's not hard to build this kind of fan in logic on top of the failover
transport, but it would be cool if the failover transport was capable of
doing fan in directly.

On Thu, Jan 29, 2009 at 8:13 AM, Gary Tully <gary.tully@gmail.com> wrote:

> have you looked at the failover transport? see:
> http://activemq.apache.org/failover-transport-reference.html
> with the backup=true option, a connection to all listed brokers will
> be created so that they are in hot standby in
> the event that the first connection is dropped. In this way, failover
> can be very fast.
>
> 2009/1/29 kaykay <kaykay.unique@gmail.com>:
> >
> > This thread is old but just curious if there has been a recent update of
> this
> > w.r.t ActiveMQ 5.2 . The problem that I am trying to solve is similar
> where
> > a consumer listens to multiple brokers (as a failover redundancy issue
> > instead of listening to a single broker).
> >
> >
> > Stepan Koltsov wrote:
> >>
> >> I'm playing with 5.0.
> >>
> >> How do you think, is it hard to write new Transport that consumes from
> >> multiple brokers?
> >>
> >> S.
> >>
> >>
> >> ttmdev wrote:
> >>>
> >>> Yes, I got similar results in my consumer testing. What version of AMQ
> >>> are you using?
> >>>
> >>> Your only recourse may be to multi thread your consumer and have it
> >>> create a connection to each of the brokers.
> >>>
> >>
> >>
> >
> > --
> > View this message in context:
> http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836p21729358.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
> >
>
>
>
> --
> http://blog.garytully.com
>
> Open Source SOA
> http://FUSESource.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message