hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5670) Hadoop configurations should be read from a distributed system
Date Wed, 12 May 2010 20:53:03 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866730#action_12866730

Allen Wittenauer commented on HADOOP-5670:

 [FWIW, I'm not tied to that schema layout--it really was just a quick and dirty example.]

One of the things Owen and I talked about last year was that you should be able to use files
as a local override.  I forgot to throw that in the mix here.  heh.  Anyway, non-hadoop configs
that there is no schema definition for could be done that way.  It also solves the one-off
problems.  [Custom -D overrides for cron'd jobs.  Think block size or replication factor for

While it seems very intricate, I don't think it is any worse that what we have with XML flat
files today.  Simpler configs could have a very simple layout as well, potentially in one
object if we lay out the schema correctly.  Chances are good that more and more folks will
have more complex config requirements and the LDAP tree layout is actually MUCH easier to
deal with than XML.  [In my case, I have 3 sets of configs, one for JT/NN, one for client,
and one for compute nodes.  That's ignoring the one-off configs for specialized processes...

It is also worth pointing out that this is really intended for something beyond a simple 'play'
grid.  These types of folks already have a scale and use case for needing this level of support.

As to using generic properties...

I don't think it will work.   Instead of 1 fetch per object class, you end up with 1 fetch
per HadoopPropertyName.  If you make HadoopPropertyName multi-valued so you can have more
than one per object class, now we need to determine which value matches which name.

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
> Rather than distributing the hadoop configuration files to every data node, compute node,
etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper,

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message