hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: Hadoop Master and Slave Discovery
Date Wed, 06 Jul 2011 16:10:19 GMT
There's a long standing "ZooKeeper DNS server" jira which can be found
here, someone has already created a basic implementation:


On Wed, Jul 6, 2011 at 2:52 AM, Steve Loughran <stevel@apache.org> wrote:
> On 05/07/11 23:00, Eric Yang wrote:
>> In another project, I have implemented a bonjour beacon (jmdns) which
>> sit on the Zookeeper nodes to advertise the location of zookeeper
>> servers.  When clients start up, it will discover location of
>> zookeeper through multicast dns.  Once, the server locations are
>> resolved (ip:port and TXT records), the clients shutdown mdns
>> resolvers.  Client proceed to use the resolved list for zookeeper
>> access.  There does not seem to be cpu overhead incurred by the
>> beacon, nor the clients.  If a client could not connect to zookeeper
>> anymore, then it will start mdns resolvers to look for new list of
>> zookeeper servers.  The code for the project is located at:
>> http://github.com/macroadster/hms
>> It may be possible to use similar approach for location resolution,
>> and load rest of the config through zookeeper.
>> regards,
>> Eric
> That's interesting; I think it's more important for clients to be able to
> bind dynamically than it is for the cluster machines, as they should be
> managed anyway.
> When I was doing the hadoop-in-VM clustering stuff, I had a well-known URL
> to serve up the relevant XML file for the cluster from the JT -all it did
> was relay the request to the JT at whatever host it had been assigned. All
> the clients needed to know was the URL of the config server, and they could
> bootstrap to working against clusters whose FS and JT URLs were different
> from run to run.
> zookeeper discovery would benefit a lot of projects

View raw message