hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <eric...@gmail.com>
Subject Re: Hadoop Master and Slave Discovery
Date Thu, 07 Jul 2011 00:05:41 GMT
Did you know that almost all linux desktop system comes with avahi
pre-installed and turn on by default?  What is more interesting is
that there are thousands of those machines broadcast in large
cooperation without anyone noticing them?  I have recently built a
multicast dns browser and look into the number of machines running in
a large company environment.  The number of desktop, laptop and
printer machines running multicast dns is far exceeding 1000 machines
in the local subnet.  They are all happily working fine without
causing any issues.  Printer works fine, itune sharing from someone
else works fine.  For some reason, things tend to work better on my
side of universe. :)  Allen, if you want to get stuck on stone age
tools, I won't stop you.

regards,
Eric

On Wed, Jul 6, 2011 at 4:02 PM, Allen Wittenauer <aw@apache.org> wrote:
>
> On Jul 5, 2011, at 2:40 AM, Steve Loughran wrote:
>> 1. you could use DNS proper, by way of Bonjour/avahi. You don't need to be running
any mDNS server to support .local, and I would strongly advise against it in a large cluster
(because .local resolution puts a lot of CPU load on every server in the subnet).
>
>        +1 mDNS doesn't scale to large sizes.  The only number I've ever heard is
up to 1000 hosts (not services!) before the whole system falls apart. I don't think it was
meant to scale past like a class C subnet.
>
>        Something else to keep in mind:  a lot of network gear gets multicast really
really wrong.  There are reasons why network admins are very happy that Hadoop doesn't use
multicast.
>
>        ... and all that's before one talks about the security implications.

Mime
View raw message