hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@apache.org>
Subject Re: Hadoop Master and Slave Discovery
Date Wed, 06 Jul 2011 23:02:41 GMT

On Jul 5, 2011, at 2:40 AM, Steve Loughran wrote:
> 1. you could use DNS proper, by way of Bonjour/avahi. You don't need to be running any
mDNS server to support .local, and I would strongly advise against it in a large cluster (because
.local resolution puts a lot of CPU load on every server in the subnet).

	+1 mDNS doesn't scale to large sizes.  The only number I've ever heard is up to 1000 hosts
(not services!) before the whole system falls apart. I don't think it was meant to scale past
like a class C subnet.  

	Something else to keep in mind:  a lot of network gear gets multicast really really wrong.
 There are reasons why network admins are very happy that Hadoop doesn't use multicast.

	... and all that's before one talks about the security implications.
View raw message