activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Strachan <james.strac...@gmail.com>
Subject Re: Single consumer for multiple brokers?
Date Thu, 29 Jan 2009 17:00:31 GMT
The fanout transport kinda has most of the implementation code to do
this; it just needs to be hacked a little so that consume start/stop
commands are sent to multiple brokers and transactions/acks are
correctly sent to the broker that sent the original message. There
might be some fun and games with transactions in there mind; but for
simple-ish use cases it shouldn't be too hard to hack the
FanoutTransport if anyone fancies having a go


2009/1/29 Jim Lloyd <jlloyd@silvertailsystems.com>:
> I'm interested in a variation of this for load balancing. Assume the volume
> of data coming from publishers is very large, and you need N brokers (think
> N is 10 or more). Furthermore, you want N+1 or N+2 redundancy so that if any
> 1 or 2 brokers die, there is sufficient capacity to continue handling all of
> the traffic (after affected publishers reconnect). The brokers might all be
> behind a hardware load balancer (e.g. F5 or NetScaler) so that all
> publishers connect via a virtual IP.
>
> Now, from the consumer side, we also have many consumers. Each consumer only
> subscribes to a subset of the available topics, and we arrange via design of
> our topics such that one consumer can always keep up with the volume of data
> published on one topic. But the consumer must connect to every broker.
>
> So, we need a fan-in variation of a failover transport that connects to
> every broker and actively consumes data from all brokers. If any broker
> disconnects, the consumer would use the exponential backoff reconnect logic
> to reconnect when the broker becomes available.
>
> So, the backup=true option isn't helpful. Instead, we want something like
> "fan_in=true".
>
> It's not hard to build this kind of fan in logic on top of the failover
> transport, but it would be cool if the failover transport was capable of
> doing fan in directly.
>
> On Thu, Jan 29, 2009 at 8:13 AM, Gary Tully <gary.tully@gmail.com> wrote:
>
>> have you looked at the failover transport? see:
>> http://activemq.apache.org/failover-transport-reference.html
>> with the backup=true option, a connection to all listed brokers will
>> be created so that they are in hot standby in
>> the event that the first connection is dropped. In this way, failover
>> can be very fast.
>>
>> 2009/1/29 kaykay <kaykay.unique@gmail.com>:
>> >
>> > This thread is old but just curious if there has been a recent update of
>> this
>> > w.r.t ActiveMQ 5.2 . The problem that I am trying to solve is similar
>> where
>> > a consumer listens to multiple brokers (as a failover redundancy issue
>> > instead of listening to a single broker).
>> >
>> >
>> > Stepan Koltsov wrote:
>> >>
>> >> I'm playing with 5.0.
>> >>
>> >> How do you think, is it hard to write new Transport that consumes from
>> >> multiple brokers?
>> >>
>> >> S.
>> >>
>> >>
>> >> ttmdev wrote:
>> >>>
>> >>> Yes, I got similar results in my consumer testing. What version of AMQ
>> >>> are you using?
>> >>>
>> >>> Your only recourse may be to multi thread your consumer and have it
>> >>> create a connection to each of the brokers.
>> >>>
>> >>
>> >>
>> >
>> > --
>> > View this message in context:
>> http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836p21729358.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>> >
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source SOA
>> http://FUSESource.com
>>
>



-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://fusesource.com/

Mime
View raw message