hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan D. Cabrera" <l...@toolazydogs.com>
Subject Re: JavaSpaces (Blitz?) and hadoop - comparison?
Date Mon, 12 Mar 2007 17:33:58 GMT

On Mar 2, 2007, at 5:06 AM, Jim White wrote:

> Doğacan Güney wrote:
>
>> On 3/2/07, Dan Creswell <dan.creswell@lonecrusader.co.uk> wrote:
>>> Nigel Daley wrote:
>>> > One more difference...
>>> > ,,,
>>> > OTOH, Hadoop servers and clients are currently pre-configured with
>>> > necessary host/ports.
>>> >
>>> Indeed - I was toying with doing something about removing this
>>> pre-configuration - worthwhile?
>> ...
>> In a data center there are typically 5 chubby instances and
>> their locations are stored in DNS. A client reads their location from
>> DNS and then traverses them to locate the Chubby master. After
>> [snipped the rest]
>
> This manual configuration in Hadoop was a limitation I noticed  
> first thing and have suggested Zeroconf (dynamic multicast central- 
> server-optional service-discovering DNS) as a solution.
>
> http://www.zeroconf.org/
>
> I've haven't looked into the details yet, but it is clear to me  
> that DNS Service Discovery would work dandy for implementing auto  
> configuration for Hadoop.
>
> http://www.dns-sd.org/
>
> Multicast also has good potential to enhance Hadoop networking  
> performance, but that is a separate issue.
>
> And for those concerned that Zeroconf is something exotic, it turns  
> out Amazon EC2 is already using it for the parameterized launch.   
> That's where the funny "169.254.169.254" address comes from.
>
> http://docs.amazonwebservices.com/AmazonEC2/dg/2006-10-01/ 
> TechnicalFAQ.html#d0e14061
>
> A few notes I've collected on Zeroconf:
>
> http://www.ifcx.org/wiki/LocalNetworking.html

There's another protocol that is similar called Service Location  
Protocol, SLP.  Thoughts?


Regards,
Alan




Mime
View raw message