hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <eric...@gmail.com>
Subject Re: Hadoop Master and Slave Discovery
Date Wed, 06 Jul 2011 18:25:44 GMT
I am currently working on RPM packages for Zookeeper, Pig, Hive and
HCat.  It may take a while for me to circle back to this.  Never the
less, it is interesting work that I would like to contrib.  Thanks

regards,
Eric

On Wed, Jul 6, 2011 at 9:18 AM, Patrick Hunt <phunt@apache.org> wrote:
> Eric, I'd be happy to work with you to get it committed if you'd like
> to take a whack. Would be a great addition to contrib.
>
> Patrick
>
> On Wed, Jul 6, 2011 at 9:15 AM, Eric Yang <eric818@gmail.com> wrote:
>> It would be nicer, if it was written in Java.  I think something wrap
>> on top of jmdns would be a better fit for Zookeeper.
>>
>> regards,
>> Eric
>>
>> On Wed, Jul 6, 2011 at 9:10 AM, Patrick Hunt <phunt@apache.org> wrote:
>>> There's a long standing "ZooKeeper DNS server" jira which can be found
>>> here, someone has already created a basic implementation:
>>> https://issues.apache.org/jira/browse/ZOOKEEPER-703
>>>
>>> Patrick
>>>
>>> On Wed, Jul 6, 2011 at 2:52 AM, Steve Loughran <stevel@apache.org> wrote:
>>>> On 05/07/11 23:00, Eric Yang wrote:
>>>>>
>>>>> In another project, I have implemented a bonjour beacon (jmdns) which
>>>>> sit on the Zookeeper nodes to advertise the location of zookeeper
>>>>> servers.  When clients start up, it will discover location of
>>>>> zookeeper through multicast dns.  Once, the server locations are
>>>>> resolved (ip:port and TXT records), the clients shutdown mdns
>>>>> resolvers.  Client proceed to use the resolved list for zookeeper
>>>>> access.  There does not seem to be cpu overhead incurred by the
>>>>> beacon, nor the clients.  If a client could not connect to zookeeper
>>>>> anymore, then it will start mdns resolvers to look for new list of
>>>>> zookeeper servers.  The code for the project is located at:
>>>>>
>>>>> http://github.com/macroadster/hms
>>>>>
>>>>> It may be possible to use similar approach for location resolution,
>>>>> and load rest of the config through zookeeper.
>>>>>
>>>>> regards,
>>>>> Eric
>>>>>
>>>>
>>>> That's interesting; I think it's more important for clients to be able to
>>>> bind dynamically than it is for the cluster machines, as they should be
>>>> managed anyway.
>>>>
>>>> When I was doing the hadoop-in-VM clustering stuff, I had a well-known URL
>>>> to serve up the relevant XML file for the cluster from the JT -all it did
>>>> was relay the request to the JT at whatever host it had been assigned. All
>>>> the clients needed to know was the URL of the config server, and they could
>>>> bootstrap to working against clusters whose FS and JT URLs were different
>>>> from run to run.
>>>>
>>>> zookeeper discovery would benefit a lot of projects
>>>>
>>>>
>>>
>>
>

Mime
View raw message