activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: Single consumer for multiple brokers?
Date Thu, 29 Jan 2009 17:30:05 GMT
http://activemq.apache.org/fanout-transport-reference.html - is the  
place to look to get started
On 29 Jan 2009, at 17:00, James Strachan wrote:

> The fanout transport kinda has most of the implementation code to do
> this; it just needs to be hacked a little so that consume start/stop
> commands are sent to multiple brokers and transactions/acks are
> correctly sent to the broker that sent the original message. There
> might be some fun and games with transactions in there mind; but for
> simple-ish use cases it shouldn't be too hard to hack the
> FanoutTransport if anyone fancies having a go
>
>
> 2009/1/29 Jim Lloyd <jlloyd@silvertailsystems.com>:
>> I'm interested in a variation of this for load balancing. Assume  
>> the volume
>> of data coming from publishers is very large, and you need N  
>> brokers (think
>> N is 10 or more). Furthermore, you want N+1 or N+2 redundancy so  
>> that if any
>> 1 or 2 brokers die, there is sufficient capacity to continue  
>> handling all of
>> the traffic (after affected publishers reconnect). The brokers  
>> might all be
>> behind a hardware load balancer (e.g. F5 or NetScaler) so that all
>> publishers connect via a virtual IP.
>>
>> Now, from the consumer side, we also have many consumers. Each  
>> consumer only
>> subscribes to a subset of the available topics, and we arrange via  
>> design of
>> our topics such that one consumer can always keep up with the  
>> volume of data
>> published on one topic. But the consumer must connect to every  
>> broker.
>>
>> So, we need a fan-in variation of a failover transport that  
>> connects to
>> every broker and actively consumes data from all brokers. If any  
>> broker
>> disconnects, the consumer would use the exponential backoff  
>> reconnect logic
>> to reconnect when the broker becomes available.
>>
>> So, the backup=true option isn't helpful. Instead, we want  
>> something like
>> "fan_in=true".
>>
>> It's not hard to build this kind of fan in logic on top of the  
>> failover
>> transport, but it would be cool if the failover transport was  
>> capable of
>> doing fan in directly.
>>
>> On Thu, Jan 29, 2009 at 8:13 AM, Gary Tully <gary.tully@gmail.com>  
>> wrote:
>>
>>> have you looked at the failover transport? see:
>>> http://activemq.apache.org/failover-transport-reference.html
>>> with the backup=true option, a connection to all listed brokers will
>>> be created so that they are in hot standby in
>>> the event that the first connection is dropped. In this way,  
>>> failover
>>> can be very fast.
>>>
>>> 2009/1/29 kaykay <kaykay.unique@gmail.com>:
>>>>
>>>> This thread is old but just curious if there has been a recent  
>>>> update of
>>> this
>>>> w.r.t ActiveMQ 5.2 . The problem that I am trying to solve is  
>>>> similar
>>> where
>>>> a consumer listens to multiple brokers (as a failover redundancy  
>>>> issue
>>>> instead of listening to a single broker).
>>>>
>>>>
>>>> Stepan Koltsov wrote:
>>>>>
>>>>> I'm playing with 5.0.
>>>>>
>>>>> How do you think, is it hard to write new Transport that  
>>>>> consumes from
>>>>> multiple brokers?
>>>>>
>>>>> S.
>>>>>
>>>>>
>>>>> ttmdev wrote:
>>>>>>
>>>>>> Yes, I got similar results in my consumer testing. What version 

>>>>>> of AMQ
>>>>>> are you using?
>>>>>>
>>>>>> Your only recourse may be to multi thread your consumer and  
>>>>>> have it
>>>>>> create a connection to each of the brokers.
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>> http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836p21729358.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> http://blog.garytully.com
>>>
>>> Open Source SOA
>>> http://FUSESource.com
>>>
>>
>
>
>
> -- 
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://fusesource.com/


Mime
View raw message