hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Capriolo (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5670) Hadoop configurations should be read from a distributed system
Date Tue, 16 Feb 2010 16:57:31 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834309#action_12834309

Edward Capriolo commented on HADOOP-5670:

I know this is not relevant to the ticket to dynamically/remotely read configuration files
but I wanted some people interested in hadoop configuration to take a look at:


The concept is to give users a IOS like interface to configure hadoop. The configurations
are generated locally and then pushed out over SSH/SCP.

The concept I am using: In most cases large portions of the configuration are shared. Their
are very few exceptions to this. (IE one node has 9 disks not 8). These exceptions are done
with an override list per host.

node6.hdfs-site.xml = shared.hdfs-site.xml + override.get(node6).hdfs-site.xml

Getting the effective configuration for a host is as simple as taking the shared list, and
replacing the overridden variables for that host. I am using a few simple abstractions, and
Java XML beans serialization for the configuration. Thus, the entire configuration is easily
dumped to a single XML file, so one File object represents the state of all of the clusters

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
> Rather than distributing the hadoop configuration files to every data node, compute node,
etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper,

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message