hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Capriolo (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5670) Hadoop configurations should be read from a distributed system
Date Tue, 16 Feb 2010 16:57:31 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834309#action_12834309
] 

Edward Capriolo commented on HADOOP-5670:
-----------------------------------------

I know this is not relevant to the ticket to dynamically/remotely read configuration files
but I wanted some people interested in hadoop configuration to take a look at:

http://www.jointhegrid.com/acod/

The concept is to give users a IOS like interface to configure hadoop. The configurations
are generated locally and then pushed out over SSH/SCP.

The concept I am using: In most cases large portions of the configuration are shared. Their
are very few exceptions to this. (IE one node has 9 disks not 8). These exceptions are done
with an override list per host.

{noformat}
node6.hdfs-site.xml = shared.hdfs-site.xml + override.get(node6).hdfs-site.xml
{noformat}

Getting the effective configuration for a host is as simple as taking the shared list, and
replacing the overridden variables for that host. I am using a few simple abstractions, and
Java XML beans serialization for the configuration. Thus, the entire configuration is easily
dumped to a single XML file, so one File object represents the state of all of the clusters
configurations.

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node,
etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper,
whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message