activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: Cluster transport ...
Date Mon, 19 May 2008 17:17:19 GMT
Agreed.

We've got a pluggable discovery mechanism; both for the broker to
advertise itself or for brokers/clients to discover things, so I guess
DNS just becomes another provider.
http://activemq.apache.org/discovery.html

The difference with DNS is you may use your existing DNS provider or
might wanna run a custom implementation to get the timeout stuff etc?

2008/5/19 Sridhar2008 <activemq@komandur.com>:
>
>
> Thanks for the link. I quickly glanced through the dnsjava website, and not
> clear if it requires a specific server for updates to work.
>
> Also, most large organizations have their own legacy DNS in place, with some
> home grown way of updating the DNS records. So we need to accommodate this
> use case (for example, making 'auto-registration' optional).
>
>
> James.Strachan wrote:
>>
>> Ah cool thanks for that! I was wondering if an existing DNS library
>> might be useful. e.g.
>>
>> http://www.dnsjava.org/
>>
>> which has a DNS client and server.
>>
>> I guess ZeroConf is kinda similar too; just not requiring a new or
>> custom DNS server (though at the expense of requiring multicast
>> support)
>>
>> 2008/5/19 Sridhar2008 <activemq@komandur.com>:
>>>> Incidentally I'm intrigued by the DNS discovery stuff; it sounds >great
>>>>:). What did you have in mind? That each broker on startup >would
>>>>register a DNS entry (with a timeout maybe so they get >removed) and
>>>
>>> Yes, auto-registration.  Couple of things to note:
>>>
>>> - The 'timeout feature' again is DNS master specific, the existing ones
>>> need
>>> to be enhanced to support this aging out. The standard timeout you see in
>>> the DNS entry is for a different purpose: refresh between  master and
>>> slave
>>> DNS servers.
>>>
>>> - There is no standard APY for web dns  registration, so we will  define
>>> an
>>> APY, with specific implementatioin to interact with the DNS server (folks
>>> can chose to change the implementation for their env).
>>>
>>> - Most messaging systems  run in trusted env, but some use cases may need
>>> authentication before a broker is allowed to do 'auto-registration'. This
>>> can be addressed later.
>>>
>>>>clients would ping DNS when attempting to connect in the
>>>>FailoverTransport?
>>>
>>> Yes, we would specify the cluster name in client config and they would
>>> connect using 'ClusterTransport' :-)
>>>
>>>
>>> James.Strachan wrote:
>>>>
>>>> 2008/5/15 Sridhar2008 <activemq@komandur.com>:
>>>>>
>>>>>
>>>>> Rob/James,
>>>>>
>>>>> Thanks for the feedback. I will address both of your questions in a
>>>>> combined
>>>>> way  :-)
>>>>>
>>>>> Rob> Do you need persistent messaging - or non-persistent only?
>>>>> James> or just pick one broker per operation/transaction?
>>>>>
>>>>> Initially, it is going to be the above. Since the messaging usage
>>>>> scenarios
>>>>> are many, I will just assume the use case of interest for now is
>>>>> durable
>>>>> transfer using cluster for high availability.  In this case I am not
>>>>> sure
>>>>> that it buys much to send to multiple brokers simultaneously (see my
>>>>> next
>>>>> paragraph below) - adds additional headaches as James sites below
>>>>> (however,
>>>>> there may be other use cases where this might be useful), in addition
>>>>> to
>>>>> making client-side logic complex.
>>>>
>>>> Agreed.
>>>>
>>>> For sending to a topic (outside of a transaction which may include
>>>> other operations) then sending the message to all brokers is a no
>>>> brainer. I guess so long as acknowledgements only get sent to the
>>>> broker they came from & for queue sending we only send to one of the
>>>> brokers it should be fairly straight forward.
>>>>
>>>> (So a little bit of hacking to the FanOutTransport should do the trick I
>>>> think).
>>>>
>>>> We could get clever going forward; where rather than randomly picking
>>>> one of the brokers (or round robbin) we kinda partition destinations
>>>> across the available brokers?
>>>>
>>>>
>>>>
>>>>> Some thoughts: Eventual goal is a broker down in the cluster implies
>>>>> capacity hit and not a service hit. So, I will rather solve this use
>>>>> case
>>>>> as
>>>>> a distributed storage issue (after 'cluster transport' is added) -
>>>>> investigate either a DHT based solution or perhaps something like
>>>>> Hadoop
>>>>> with a JDBC interface. The state is replicated in multiple nodes of the
>>>>> cluster so the broker that is down can be ignored (ie no need to sweat
>>>>> on
>>>>> recovering its state).
>>>>
>>>> Agreed.
>>>>
>>>> FWIW distributing the state is trivial - the FanOutTransport can do
>>>> this today really. The issue is ensuring consistency (so more to do
>>>> with locking & consistency than moving the messages around).
>>>>
>>>> Incidentally I'm intrigued by the DNS discovery stuff; it sounds great
>>>> :). What did you have in mind? That each broker on startup would
>>>> register a DNS entry (with a timeout maybe so they get removed) and
>>>> clients would ping DNS when attempting to connect in the
>>>> FailoverTransport?
>>>>
>>>> --
>>>> James
>>>> -------
>>>> http://macstrac.blogspot.com/
>>>>
>>>> Open Source Integration
>>>> http://open.iona.com
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17323314.html
>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> James
>> -------
>> http://macstrac.blogspot.com/
>>
>> Open Source Integration
>> http://open.iona.com
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17323601.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>



-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com

Mime
View raw message