hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Hunt (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5670) Hadoop configurations should be read from a distributed system
Date Tue, 06 Apr 2010 16:01:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854033#action_12854033

Patrick Hunt commented on HADOOP-5670:

Allen and I had a conversation about this a while ago and he made some very good points for
storing certain data
in a DS vs ZK. In particular user data makes alot of sense imo to be stored in DS. Data. Keep
in mind that
ZK is all about coordination, not "data storage". We don't support search for example, which
is a significant 
feature in most DSs. Also integration with legacy systems (your existing user database) is
also a feature
of most DSs that ZK does not have. While ZK could do these things, a typical
DS will do them for you out of the box, and make your admin's lives easier in the sense that
they already
have experience with this.

At the same time things like coordination are best served by ZK. Keeping track of which nodes
are allocated
to which functions, the status of processes and coordinating operations between them, the
load and activity 
of processes (nodes), Leader election within a highly reliable/available service, distributed
locks and work
queues, etc...

Take a look at LinkedIn's Norbert for an example of one instantiation of something like this:

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
> Rather than distributing the hadoop configuration files to every data node, compute node,
etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper,

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message