hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5670) Hadoop configurations should be read from a distributed system
Date Thu, 23 Apr 2009 13:23:30 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701928#action_12701928
] 

Steve Loughran commented on HADOOP-5670:
----------------------------------------

# Apache Directory implements an NFS filesystem front end to LDAP data, so you could *today*
use it to provide all of the configuration data in a single point of failure/HA directory
service
# I do subclass Configuration for my work, with a ManagedConfiguration being read from SmartFrog,
rather than the filesystem. I have to pass an instance of this down when any service gets
created. 
# These Configurations get serialised and sent around with Job submissions, so they soon become
static snapshots of configuration when jobs were queued, not when they were executed. You
can't be dynamic without changing this behaviour.
# There are 350+ places in the Hadoop codebase where something calls the {{new Configuration()}
operation, to create a config purely from the registered files. This is bad; some kind of
factory approach would be better, with the factory settable on a per-JVM or per-thread basis.
# HADOOP-3582 collects other requirements for configuration.

I am willing to get involved in this, as config is part of management, and I probably have
the most dynamic configuration setup to date. But I'd rather wait until after the service
lifecycle stuff is in, so change can be handled more gradually. If any configuration changes
were to go into 0.21, I'd opt for a factory mechanism for creating new Configuration instances,
so that people exploring configuration options can control things better.

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node,
etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper,
whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message