hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@maprtech.com>
Subject Re: Hadoop Master and Slave Discovery
Date Mon, 04 Jul 2011 17:22:02 GMT
One reasonable suggestion that I have heard recently was to do like Google
does and put a DNS front end onto Zookeeper.  Machines would need to have
DNS set up properly and a requests for a special ZK based domain would have
to be delegated to the fancy DNS setup, but this would allow all kinds of
host targeted configuration settings to be moved by a very standardized
network protocol.  There are difficulties moving port numbers and all you
really get are hostnames, but it is a nice trick since configuration of
which nameserver to use is a common admin task.

On Mon, Jul 4, 2011 at 3:44 AM, Steve Loughran <stevel@apache.org> wrote:

> On 03/07/11 03:11, Raja Nagendra Kumar wrote:
>> Hi,
>> Instead of depending on local syncup to configuration files, would it be a
>> nice way to adopt JINI Discovery model, where in masters and slaves can
>> discover each other dynamically through a UDP broadcast/heart beat methods
> That assumes that UDP Broadcast is supported through the switches (many
> turn it off as it creates too much traffic), or UDP multicast is supported
> (as an example of an infrastructure that does not, play with EC2)
>  This would mean, any machine can come up and say I am a slave and
>> automatically discover the master and start supporting the master with in<
>> x seconds.
> How will your slave determine that the master that it has bonded to is the
> master that it should bond to and not something malicious within the same
> multicast range? It's possible, but you have generally have to have
> configuration files in the worker nodes.
> There's nothing to stop your Configuration Management layer using discovery
> or central config servers (Zookeeper, Anubis, LDAP, DNS, ...), which then
> pushes desired state information to the client nodes. These can deal with
> auth and may support infrastructures that don't support broadcast or
> multicast. Such tooling also gives you the ability to push out host table
> config, JVM options, logging parameters, and bounce worker nodes into new
> states without waiting for them to timeout and try to rediscover new
> masters.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message