activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sridhar2008 <activ...@komandur.com>
Subject Re: Cluster transport ...
Date Mon, 19 May 2008 17:25:04 GMT

>Agreed.
>
>We've got a pluggable discovery mechanism; both for the broker >to
>advertise itself or for brokers/clients to discover things, so I >guess
>DNS just becomes another provider.
>http://activemq.apache.org/discovery.html

Exactly ! The pluggable nature of ActiveMQ is great for enhancements like
these.

>The difference with DNS is you may use your existing DNS >provider or
>might wanna run a custom implementation to get the timeout >stuff etc?

Again, right on ! The timeout may not be that big a deal depending on the
implementation ... for example, it might be a lazy cleanup process which may
or may not be part of the DNS master server. For the clients (ie producers
or consumers), if a broker in the list does not respond, it will just
timeout to the next (note: 'next' could be as simple as next in the list or
based on some policy).


James.Strachan wrote:
> 
> Agreed.
> 
> We've got a pluggable discovery mechanism; both for the broker to
> advertise itself or for brokers/clients to discover things, so I guess
> DNS just becomes another provider.
> http://activemq.apache.org/discovery.html
> 
> The difference with DNS is you may use your existing DNS provider or
> might wanna run a custom implementation to get the timeout stuff etc?
> 
> 2008/5/19 Sridhar2008 <activemq@komandur.com>:
>>
>>
>> Thanks for the link. I quickly glanced through the dnsjava website, and
>> not
>> clear if it requires a specific server for updates to work.
>>
>> Also, most large organizations have their own legacy DNS in place, with
>> some
>> home grown way of updating the DNS records. So we need to accommodate
>> this
>> use case (for example, making 'auto-registration' optional).
>>
>>
>> James.Strachan wrote:
>>>
>>> Ah cool thanks for that! I was wondering if an existing DNS library
>>> might be useful. e.g.
>>>
>>> http://www.dnsjava.org/
>>>
>>> which has a DNS client and server.
>>>
>>> I guess ZeroConf is kinda similar too; just not requiring a new or
>>> custom DNS server (though at the expense of requiring multicast
>>> support)
>>>
>>> 2008/5/19 Sridhar2008 <activemq@komandur.com>:
>>>>> Incidentally I'm intrigued by the DNS discovery stuff; it sounds
>>>>> >great
>>>>>:). What did you have in mind? That each broker on startup >would
>>>>>register a DNS entry (with a timeout maybe so they get >removed) and
>>>>
>>>> Yes, auto-registration.  Couple of things to note:
>>>>
>>>> - The 'timeout feature' again is DNS master specific, the existing ones
>>>> need
>>>> to be enhanced to support this aging out. The standard timeout you see
>>>> in
>>>> the DNS entry is for a different purpose: refresh between  master and
>>>> slave
>>>> DNS servers.
>>>>
>>>> - There is no standard APY for web dns  registration, so we will 
>>>> define
>>>> an
>>>> APY, with specific implementatioin to interact with the DNS server
>>>> (folks
>>>> can chose to change the implementation for their env).
>>>>
>>>> - Most messaging systems  run in trusted env, but some use cases may
>>>> need
>>>> authentication before a broker is allowed to do 'auto-registration'.
>>>> This
>>>> can be addressed later.
>>>>
>>>>>clients would ping DNS when attempting to connect in the
>>>>>FailoverTransport?
>>>>
>>>> Yes, we would specify the cluster name in client config and they would
>>>> connect using 'ClusterTransport' :-)
>>>>
>>>>
>>>> James.Strachan wrote:
>>>>>
>>>>> 2008/5/15 Sridhar2008 <activemq@komandur.com>:
>>>>>>
>>>>>>
>>>>>> Rob/James,
>>>>>>
>>>>>> Thanks for the feedback. I will address both of your questions in
a
>>>>>> combined
>>>>>> way  :-)
>>>>>>
>>>>>> Rob> Do you need persistent messaging - or non-persistent only?
>>>>>> James> or just pick one broker per operation/transaction?
>>>>>>
>>>>>> Initially, it is going to be the above. Since the messaging usage
>>>>>> scenarios
>>>>>> are many, I will just assume the use case of interest for now is
>>>>>> durable
>>>>>> transfer using cluster for high availability.  In this case I am
not
>>>>>> sure
>>>>>> that it buys much to send to multiple brokers simultaneously (see
my
>>>>>> next
>>>>>> paragraph below) - adds additional headaches as James sites below
>>>>>> (however,
>>>>>> there may be other use cases where this might be useful), in addition
>>>>>> to
>>>>>> making client-side logic complex.
>>>>>
>>>>> Agreed.
>>>>>
>>>>> For sending to a topic (outside of a transaction which may include
>>>>> other operations) then sending the message to all brokers is a no
>>>>> brainer. I guess so long as acknowledgements only get sent to the
>>>>> broker they came from & for queue sending we only send to one of
the
>>>>> brokers it should be fairly straight forward.
>>>>>
>>>>> (So a little bit of hacking to the FanOutTransport should do the trick
>>>>> I
>>>>> think).
>>>>>
>>>>> We could get clever going forward; where rather than randomly picking
>>>>> one of the brokers (or round robbin) we kinda partition destinations
>>>>> across the available brokers?
>>>>>
>>>>>
>>>>>
>>>>>> Some thoughts: Eventual goal is a broker down in the cluster implies
>>>>>> capacity hit and not a service hit. So, I will rather solve this
use
>>>>>> case
>>>>>> as
>>>>>> a distributed storage issue (after 'cluster transport' is added)
-
>>>>>> investigate either a DHT based solution or perhaps something like
>>>>>> Hadoop
>>>>>> with a JDBC interface. The state is replicated in multiple nodes
of
>>>>>> the
>>>>>> cluster so the broker that is down can be ignored (ie no need to
>>>>>> sweat
>>>>>> on
>>>>>> recovering its state).
>>>>>
>>>>> Agreed.
>>>>>
>>>>> FWIW distributing the state is trivial - the FanOutTransport can do
>>>>> this today really. The issue is ensuring consistency (so more to do
>>>>> with locking & consistency than moving the messages around).
>>>>>
>>>>> Incidentally I'm intrigued by the DNS discovery stuff; it sounds great
>>>>> :). What did you have in mind? That each broker on startup would
>>>>> register a DNS entry (with a timeout maybe so they get removed) and
>>>>> clients would ping DNS when attempting to connect in the
>>>>> FailoverTransport?
>>>>>
>>>>> --
>>>>> James
>>>>> -------
>>>>> http://macstrac.blogspot.com/
>>>>>
>>>>> Open Source Integration
>>>>> http://open.iona.com
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17323314.html
>>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> James
>>> -------
>>> http://macstrac.blogspot.com/
>>>
>>> Open Source Integration
>>> http://open.iona.com
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17323601.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> Open Source Integration
> http://open.iona.com
> 
> 

-- 
View this message in context: http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17323803.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.


Mime
View raw message