hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Hadoop Master and Slave Discovery
Date Mon, 04 Jul 2011 10:44:23 GMT
On 03/07/11 03:11, Raja Nagendra Kumar wrote:
> Hi,
> Instead of depending on local syncup to configuration files, would it be a
> nice way to adopt JINI Discovery model, where in masters and slaves can
> discover each other dynamically through a UDP broadcast/heart beat methods

That assumes that UDP Broadcast is supported through the switches (many 
turn it off as it creates too much traffic), or UDP multicast is 
supported (as an example of an infrastructure that does not, play with EC2)

> This would mean, any machine can come up and say I am a slave and
> automatically discover the master and start supporting the master with in<
> x seconds.

How will your slave determine that the master that it has bonded to is 
the master that it should bond to and not something malicious within the 
same multicast range? It's possible, but you have generally have to have 
configuration files in the worker nodes.

There's nothing to stop your Configuration Management layer using 
discovery or central config servers (Zookeeper, Anubis, LDAP, DNS, ...), 
which then pushes desired state information to the client nodes. These 
can deal with auth and may support infrastructures that don't support 
broadcast or multicast. Such tooling also gives you the ability to push 
out host table config, JVM options, logging parameters, and bounce 
worker nodes into new states without waiting for them to timeout and try 
to rediscover new masters.

View raw message