hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Creswell <dan.cresw...@lonecrusader.co.uk>
Subject Re: JavaSpaces (Blitz?) and hadoop - comparison?
Date Thu, 15 Mar 2007 10:50:33 GMT
Alan D. Cabrera wrote:
>
> On Mar 2, 2007, at 5:06 AM, Jim White wrote:
>
>> Doğacan Güney wrote:
>>
>>> On 3/2/07, Dan Creswell <dan.creswell@lonecrusader.co.uk> wrote:
>>>> Nigel Daley wrote:
>>>> > One more difference...
>>>> > ,,,
>>>> > OTOH, Hadoop servers and clients are currently pre-configured with
>>>> > necessary host/ports.
>>>> >
>>>> Indeed - I was toying with doing something about removing this
>>>> pre-configuration - worthwhile?
>>> ...
>>> In a data center there are typically 5 chubby instances and
>>> their locations are stored in DNS. A client reads their location from
>>> DNS and then traverses them to locate the Chubby master. After
>>> [snipped the rest]
>>
>> This manual configuration in Hadoop was a limitation I noticed first
>> thing and have suggested Zeroconf (dynamic multicast
>> central-server-optional service-discovering DNS) as a solution.
>>
>> http://www.zeroconf.org/
>>
>> I've haven't looked into the details yet, but it is clear to me that
>> DNS Service Discovery would work dandy for implementing auto
>> configuration for Hadoop.
>>
>> http://www.dns-sd.org/
>>
>> Multicast also has good potential to enhance Hadoop networking
>> performance, but that is a separate issue.
>>
>> And for those concerned that Zeroconf is something exotic, it turns
>> out Amazon EC2 is already using it for the parameterized launch. 
>> That's where the funny "169.254.169.254" address comes from.
>>
>> http://docs.amazonwebservices.com/AmazonEC2/dg/2006-10-01/TechnicalFAQ.html#d0e14061
>>
>>
>> A few notes I've collected on Zeroconf:
>>
>> http://www.ifcx.org/wiki/LocalNetworking.html
>
> There's another protocol that is similar called Service Location
> Protocol, SLP.  Thoughts?
>

Looks an awful lot like the Jini service location protocol - seems like
it got published out of Sun in a similar timeframe too.

I'm somewhat biased but I'd be looking at SLP and/or Jini although
ZeroConf might be a quicker initial hack.  Not sure whether ZeroConf'd
be the right solution over the long term or with larger clusters -
anybody got experience there?

Dan.


Mime
View raw message