hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim White <...@pagesmiths.com>
Subject Re: JavaSpaces (Blitz?) and hadoop - comparison?
Date Fri, 02 Mar 2007 13:06:44 GMT
Doğacan Güney wrote:

> On 3/2/07, Dan Creswell <dan.creswell@lonecrusader.co.uk> wrote:
> 
>> Nigel Daley wrote:
>> > One more difference...
>> > ,,,
>> > OTOH, Hadoop servers and clients are currently pre-configured with
>> > necessary host/ports.
>> >
>> Indeed - I was toying with doing something about removing this
>> pre-configuration - worthwhile?
> 
> ...
> In a data center there are typically 5 chubby instances and
> their locations are stored in DNS. A client reads their location from
> DNS and then traverses them to locate the Chubby master. After
> [snipped the rest]

This manual configuration in Hadoop was a limitation I noticed first 
thing and have suggested Zeroconf (dynamic multicast 
central-server-optional service-discovering DNS) as a solution.

http://www.zeroconf.org/

I've haven't looked into the details yet, but it is clear to me that DNS 
Service Discovery would work dandy for implementing auto configuration 
for Hadoop.

http://www.dns-sd.org/

Multicast also has good potential to enhance Hadoop networking 
performance, but that is a separate issue.

And for those concerned that Zeroconf is something exotic, it turns out 
Amazon EC2 is already using it for the parameterized launch.  That's 
where the funny "169.254.169.254" address comes from.

http://docs.amazonwebservices.com/AmazonEC2/dg/2006-10-01/TechnicalFAQ.html#d0e14061

A few notes I've collected on Zeroconf:

http://www.ifcx.org/wiki/LocalNetworking.html

Jim


Mime
View raw message