activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sridhar2008 <activ...@komandur.com>
Subject Re: Cluster transport ...
Date Thu, 15 May 2008 05:17:34 GMT


Rob/James,

Thanks for the feedback. I will address both of your questions in a combined
way  :-)

Rob> Do you need persistent messaging - or non-persistent only?
James> or just pick one broker per operation/transaction?

Initially, it is going to be the above. Since the messaging usage scenarios
are many, I will just assume the use case of interest for now is durable
transfer using cluster for high availability.  In this case I am not sure
that it buys much to send to multiple brokers simultaneously (see my next
paragraph below) - adds additional headaches as James sites below (however,
there may be other use cases where this might be useful), in addition to
making client-side logic complex. 

Some thoughts: Eventual goal is a broker down in the cluster implies
capacity hit and not a service hit. So, I will rather solve this use case as
a distributed storage issue (after 'cluster transport' is added) -
investigate either a DHT based solution or perhaps something like Hadoop
with a JDBC interface. The state is replicated in multiple nodes of the
cluster so the broker that is down can be ignored (ie no need to sweat on
recovering its state).

Thanks,
Regards,
- Sridhar


James.Strachan wrote:
> 
> Sridhar - this all sounds great stuff!
> 
> As Rob said, on the client side we could hack the fanout transport a
> bit more to do much of that - communicating to multiple brokers and so
> forth. Are you intending the client to replicate each operation on
> each broker; or just pick one broker per operation/transaction? If its
> the latter its pretty trivial to do - if its the former it opens up
> some interesting challenges in ensuring queue ordering and dispatching
> maintains consistency across the replicas - e.g. electing one broker
> the master and then having broker-broker synchronization when a master
> fails etc.
> 
> Also the DNS discovery system sounds great and should benefit all
> ActiveMQ users - whether they use this particular clustering model or
> not.
> 
> So I guess in terms of ease-of-completing, DNS discovery and client
> load balancing across a cluster of brokers are both great pieces of
> work that shouldn't be too hard to get coded, tested and documented.
> 
> Having the clients replicate their operations across the cluster of
> brokers and having some master/slave/broker-sync mechanism is gonna
> take much more meaty discussions to work out the details - but that
> sounds great fun too :).
> 
> The tricky bit is gonna be the broker-broker synchronization I'd
> imagine - such as wh
> 2008/5/14 Rob Davies <rajdavies@gmail.com>:
>>
>> On 14 May 2008, at 01:27, Sridhar2008 wrote:
>>
>>>
>>>
>>> Hi,
>>>
>>> I'm working on a feature needed for our internal use for a  highly
>>> available
>>> messaging bus service. Per James, I am capturing  a short description of
>>> our
>>> needs. I would appreciate any feedback, for the larger activemq user
>>> community usage ...
>>>
>>>       +----B1 ---+
>>>        |                |
>>>  P---+---B2-----+---- C
>>>        |                |
>>>        +---B3-----+
>>>
>>> We'd like producers/consumers to be  connected to multiple brokers
>>> simultaneously for high availability - we have had  outages due to 
>>> single
>>> broker failure and we also the desire seamless in/out of servicing
>>> brokers
>>> as needed or based on jmx monitoring.  In addition, producers and
>>> consumers
>>> will not be hardcoded with specific broker addresses. They will have the
>>> broker cluster 'cname' which is resolved as  'service record' in the DNS
>>> ---
>>> so we will also be contributing a new DNS based discovery module. This
>>> facilitates dynamic addition/removal of brokers as needed for :
>>> capacity,
>>> availability, etc. The producer will send to all the available brokers
>>> (perhaps round robin) and consumers will accept from all the available
>>> brokers.
>>>
>>> This is the basic premise of the feature. This can be enhanced in
>>> various
>>> ways. For example, to take a broker out of service, I plan on
>>> configuring
>>> through the jmx, so the broker will stop accepting new messages, while
>>> the
>>> consumers are draining existing messages. Once the messages are drained,
>>> it
>>> will shut itself down. Another possible enhancement is to load balance
>>> messages among the brokers.
>>>
>>> Thanks,
>>> Regards,
>>> - Sridhar Komandur
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17221068.html
>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>>
>>
>> Hi Sridhar,
>>
>> this looks like you'll be using the fanout transport -
>> http://activemq.apache.org/fanout-transport-reference.html
>> Do you need persistent messaging - or non-persistent only?
>>
>> The DNS discovery module looks interesting!
>>
>>
>>
>>
>> cheers,
>>
>> Rob
>>
>> http://open.iona.com/products/enterprise-activemq
>> http://rajdavies.blogspot.com/
>>
>>
>>
>>
>>
> 
> 
> 
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> Open Source Integration
> http://open.iona.com
> 
> 

-- 
View this message in context: http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17246093.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.


Mime
View raw message