hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3582) Improve how Hadoop gets configured
Date Thu, 12 Feb 2009 14:09:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672980#action_12672980

Steve Loughran commented on HADOOP-3582:

One think to look out for here is the places in the code, such as hdfs.protocol.FSConstants,
that do things like
public static final int BUFFER_SIZE = new Configuration().getInt("io.file.buffer.size", 4096)

That is, set a constant value from a default configuration. Currently, the only way to change
such a config is to place a new hadoop-site.xml on the CP before the class is initialized.

To have some other way of managing configurations, we'd really need a factory that could be
driven by JVM properties, 

public static final int BUFFER_SIZE = ConfigurationFactory.newInstance().getInt("io.file.buffer.size",

> Improve how Hadoop gets configured
> ----------------------------------
>                 Key: HADOOP-3582
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3582
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Steve Loughran
> Improve how Hadoop gets configured
> The current approach of a two-level XML configuration file works; it offers
>  * default values in (easily overridden) configuration files, rather than just Java source
>  * A way to override the default values
>  * conversion from string to float, double.
>  * with the ${property} evaluation, there is some ability to reference system values
for some limited adaptation.
>  * errors show up at parse time (except for value parse problems)
>  * A serialization format to exchange configuration with other nodes 
>  * the possibility of updating a local (in-memory) configuration 
> But it has limits
> [1] Requires a change to the XML files to be pushed out to every node
> [2] Differences between configurations can cause obscure bugs
> [3] No support for complex hierarchical configurations
> [4] No easy way to cross-reference data other than copy and paste.
> [5] No way for a deployed instance to update configuration data for other instances to
> [6] Value type checking/dereferencing failure is not signalled by a custom error; there
is no explicit exception on any of the get/set operations.
> [7] No consistency with names. 
> [8] Not easily managed by different configuration architectures/tools
> This issue is to group/track the problems, then discuss solutions. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message