hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-785) Divide the server and client configurations
Date Mon, 13 Aug 2007 18:01:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12519481

Arun C Murthy commented on HADOOP-785:

I'd like to assuage some concerns and clarify/change some more about certain aspects of the
above proposal, here we go...

Existing issues:
a) Easy to misconfigure (e.g. Owen's original description) and hard to debug.
b) No way to separately configure the servers and clients (both dfs & mapred clients).
c) Ease administration of large clusters, ensuring that administrators have a fair amount
of control over some configuration knobs, while at the same time ensuring users can control
*relevant" config knobs.
d) No way to override hadoop-site.xml from job.xml (e.g. turn on/off speculative execution
from job.xml).


Organise the config tree around where things in cluster are used i.e. things used only by
server, and knobs clients (both hdfs and mapred clients) may tweak.

To help clarify the above I propose we re-work the config files as follows:

a) hadoop-default.xml - Read-only defaults.
b) hadoop-site.xml - General site-specific values e.g. you'd put {{fs.default.name}} and {{mapred.job.tracker}}
here. Important change: values here *can be overridden* (code-speak: we'd do {{addDefaultResource(hadoop-site)}}
rather than {{addFinalResource(hadoop-site)}} as done today).
c) hadoop-server.xml - Hadoop servers (daemons) are configured via this, and values here *cannot*
be overridden. One would, for e.g., put {{dfs.replication}} here.
d) hadoop-client.xml - Hadoop clients (bot hdfs and mapred clients) are configured via this
file, which, again, cannot be overridden. One would, for e.g., put {{dfs.client.buffer.dir}}
e) job.xml - Job-specific configuration at run-time (as today).

Thus, hadoop-site.xml, which is the non-overridable today, is kept around as a general, site-specific,
overridable configuration file, where *any* config knob can be tweaked. We could completely
do away with hadoop-site.xml, however to keep things backward compatible and to ensure that
common configuration isn't duplicated (else, we would have to force users to specify {{fs.default.name}}/{{mapred.job.tracker}}
in both hadoop-server.xml & hadoop-client.xml, which seems redundant).

However, hadoop-server.xml and hadoop-client.xml are 2 new config files for hadoop servers
(daemons) and clients which *cannot* be overridden in the server and client context respectively.

The class heirarchy now looks like this:

  abstract public class Configuration {
    // ...

  public class ServerConfiguration extends Configuration {
    // reads hadoop-default.xml, hadoop-site.xml & hadoop-server.xml, in that order
  public class ClientConfiguration extends Configuration {
    // reads hadoop-default.xml, hadoop-site.xml & hadoop-client.xml, in that order
  public class JobConf extends ClientConfiguration {
    // reads hadoop-default.xml, hadoop-site.xml, hadoop-client.xml & job.xml in that

I propose we tighten interfaces of servers/clients to reflect that they are using a specific
*type* of configuration. E.g.

  class JobTracker {
    public JobTracker(JobConf conf) {
      // ...


  class JobTracker {
    public JobTracker(ServerConfiguration conf) {
      // ...


  public class DFSClient {
    public DFSClient(Configuration conf) {
      // ...


  public class DFSClient {
    public DFSClient(ClientConfiguration conf) {
      // ...

and so on... (Map/Reduce public apis already correctly defined in terms of JobConf already).

Thus, we would create different types of configuration objects (ServerConfiguration/ClientConfiguration/JobConf)
and use them in the relevant sub-systems (NameNode/JobTracker: ServerConfiguration, DFSClient/MapReduceBase:
ClientConfiguration/JobConf) etc.

This has the benefit that 
a) It matches the context and use cases of the designated configuration files.
b) Users have a fair amount of control over *relevant* knobs e.g. they can specify {{io.buffer.size}}
in JobConf and that will be used by, for e.g., in SequenceFile via {{InputFormat.getRecordReader(InputSplit,
JobConf, Reporter)}}.
c) Ensures that administration of the large cluster is eased and that admins have a fair amount
of control over the various configuration knobs. E.g. there is no way for a *user* to tweak
{{dfs.client.buffer.dir}} via the JobConf since DFSClient is defined in terms of ClientConfiguration
which clearly never looks at job.xml.

Also, I'd like to use this redo to go ahead and implement Doug's suggestion of using static
methods, where applicable, to setup the actual values; thereby stop dumping too many get/set
methods in Configuration/JobConf.

Last, not least, is how we go about communicating *where* the right place for each config
knob is; in this regard I guess the consensus has been to not introduce newer xml tags/attributes
and hence to use the *description* tag and place it along with the 'level' (expert/intermediate

> Divide the server and client configurations
> -------------------------------------------
>                 Key: HADOOP-785
>                 URL: https://issues.apache.org/jira/browse/HADOOP-785
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.9.0
>            Reporter: Owen O'Malley
>            Assignee: Arun C Murthy
>             Fix For: 0.15.0
> The configuration system is easy to misconfigure and I think we need to strongly divide
the server from client configs. 
> An example of the problem was a configuration where the task tracker has a hadoop-site.xml
that set mapred.reduce.tasks to 1. Therefore, the job tracker had the right number of reduces,
but the map task thought there was a single reduce. This lead to a hard to find diagnose failure.
> Therefore, I propose separating out the configuration types as:
> class Configuration;
> // reads site-default.xml, hadoop-default.xml
> class ServerConf extends Configuration;
> // reads hadoop-server.xml, $super
> class DfsServerConf extends ServerConf;
> // reads dfs-server.xml, $super
> class MapRedServerConf extends ServerConf;
> // reads mapred-server.xml, $super
> class ClientConf extends Configuration;
> // reads hadoop-client.xml, $super
> class JobConf extends ClientConf;
> // reads job.xml, $super
> Note in particular, that nothing corresponds to hadoop-site.xml, which overrides both
client and server configs. Furthermore, the properties from the *-default.xml files should
never be saved into the job.xml.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message