hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Hadoop Master and Slave Discovery
Date Wed, 06 Jul 2011 09:52:00 GMT
On 05/07/11 23:00, Eric Yang wrote:
> In another project, I have implemented a bonjour beacon (jmdns) which
> sit on the Zookeeper nodes to advertise the location of zookeeper
> servers.  When clients start up, it will discover location of
> zookeeper through multicast dns.  Once, the server locations are
> resolved (ip:port and TXT records), the clients shutdown mdns
> resolvers.  Client proceed to use the resolved list for zookeeper
> access.  There does not seem to be cpu overhead incurred by the
> beacon, nor the clients.  If a client could not connect to zookeeper
> anymore, then it will start mdns resolvers to look for new list of
> zookeeper servers.  The code for the project is located at:
> http://github.com/macroadster/hms
> It may be possible to use similar approach for location resolution,
> and load rest of the config through zookeeper.
> regards,
> Eric

That's interesting; I think it's more important for clients to be able 
to bind dynamically than it is for the cluster machines, as they should 
be managed anyway.

When I was doing the hadoop-in-VM clustering stuff, I had a well-known 
URL to serve up the relevant XML file for the cluster from the JT -all 
it did was relay the request to the JT at whatever host it had been 
assigned. All the clients needed to know was the URL of the config 
server, and they could bootstrap to working against clusters whose FS 
and JT URLs were different from run to run.

zookeeper discovery would benefit a lot of projects

View raw message