activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sridhar2008 <>
Subject Re: Cluster transport ...
Date Thu, 15 May 2008 05:17:34 GMT


Thanks for the feedback. I will address both of your questions in a combined
way  :-)

Rob> Do you need persistent messaging - or non-persistent only?
James> or just pick one broker per operation/transaction?

Initially, it is going to be the above. Since the messaging usage scenarios
are many, I will just assume the use case of interest for now is durable
transfer using cluster for high availability.  In this case I am not sure
that it buys much to send to multiple brokers simultaneously (see my next
paragraph below) - adds additional headaches as James sites below (however,
there may be other use cases where this might be useful), in addition to
making client-side logic complex. 

Some thoughts: Eventual goal is a broker down in the cluster implies
capacity hit and not a service hit. So, I will rather solve this use case as
a distributed storage issue (after 'cluster transport' is added) -
investigate either a DHT based solution or perhaps something like Hadoop
with a JDBC interface. The state is replicated in multiple nodes of the
cluster so the broker that is down can be ignored (ie no need to sweat on
recovering its state).

- Sridhar

James.Strachan wrote:
> Sridhar - this all sounds great stuff!
> As Rob said, on the client side we could hack the fanout transport a
> bit more to do much of that - communicating to multiple brokers and so
> forth. Are you intending the client to replicate each operation on
> each broker; or just pick one broker per operation/transaction? If its
> the latter its pretty trivial to do - if its the former it opens up
> some interesting challenges in ensuring queue ordering and dispatching
> maintains consistency across the replicas - e.g. electing one broker
> the master and then having broker-broker synchronization when a master
> fails etc.
> Also the DNS discovery system sounds great and should benefit all
> ActiveMQ users - whether they use this particular clustering model or
> not.
> So I guess in terms of ease-of-completing, DNS discovery and client
> load balancing across a cluster of brokers are both great pieces of
> work that shouldn't be too hard to get coded, tested and documented.
> Having the clients replicate their operations across the cluster of
> brokers and having some master/slave/broker-sync mechanism is gonna
> take much more meaty discussions to work out the details - but that
> sounds great fun too :).
> The tricky bit is gonna be the broker-broker synchronization I'd
> imagine - such as wh
> 2008/5/14 Rob Davies <>:
>> On 14 May 2008, at 01:27, Sridhar2008 wrote:
>>> Hi,
>>> I'm working on a feature needed for our internal use for a  highly
>>> available
>>> messaging bus service. Per James, I am capturing  a short description of
>>> our
>>> needs. I would appreciate any feedback, for the larger activemq user
>>> community usage ...
>>>       +----B1 ---+
>>>        |                |
>>>  P---+---B2-----+---- C
>>>        |                |
>>>        +---B3-----+
>>> We'd like producers/consumers to be  connected to multiple brokers
>>> simultaneously for high availability - we have had  outages due to 
>>> single
>>> broker failure and we also the desire seamless in/out of servicing
>>> brokers
>>> as needed or based on jmx monitoring.  In addition, producers and
>>> consumers
>>> will not be hardcoded with specific broker addresses. They will have the
>>> broker cluster 'cname' which is resolved as  'service record' in the DNS
>>> ---
>>> so we will also be contributing a new DNS based discovery module. This
>>> facilitates dynamic addition/removal of brokers as needed for :
>>> capacity,
>>> availability, etc. The producer will send to all the available brokers
>>> (perhaps round robin) and consumers will accept from all the available
>>> brokers.
>>> This is the basic premise of the feature. This can be enhanced in
>>> various
>>> ways. For example, to take a broker out of service, I plan on
>>> configuring
>>> through the jmx, so the broker will stop accepting new messages, while
>>> the
>>> consumers are draining existing messages. Once the messages are drained,
>>> it
>>> will shut itself down. Another possible enhancement is to load balance
>>> messages among the brokers.
>>> Thanks,
>>> Regards,
>>> - Sridhar Komandur
>>> --
>>> View this message in context:
>>> Sent from the ActiveMQ - Dev mailing list archive at
>> Hi Sridhar,
>> this looks like you'll be using the fanout transport -
>> Do you need persistent messaging - or non-persistent only?
>> The DNS discovery module looks interesting!
>> cheers,
>> Rob
> -- 
> James
> -------
> Open Source Integration

View this message in context:
Sent from the ActiveMQ - Dev mailing list archive at

View raw message