hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-785) Divide the server and client configurations
Date Wed, 06 Dec 2006 08:21:21 GMT
Divide the server and client configurations

                 Key: HADOOP-785
                 URL: http://issues.apache.org/jira/browse/HADOOP-785
             Project: Hadoop
          Issue Type: Improvement
          Components: conf
    Affects Versions: 0.9.0
            Reporter: Owen O'Malley
         Assigned To: Arun C Murthy
             Fix For: 0.10.0

The configuration system is easy to misconfigure and I think we need to strongly divide the
server from client configs. 

An example of the problem was a configuration where the task tracker has a hadoop-site.xml
that set mapred.reduce.tasks to 1. Therefore, the job tracker had the right number of reduces,
but the map task thought there was a single reduce. This lead to a hard to find diagnose failure.

Therefore, I propose separating out the configuration types as:

class Configuration;
// reads site-default.xml, hadoop-default.xml

class ServerConf extends Configuration;
// reads hadoop-server.xml, $super

class DfsServerConf extends ServerConf;
// reads dfs-server.xml, $super

class MapRedServerConf extends ServerConf;
// reads mapred-server.xml, $super

class ClientConf extends Configuration;
// reads hadoop-client.xml, $super

class JobConf extends ClientConf;
// reads job.xml, $super

Note in particular, that nothing corresponds to hadoop-site.xml, which overrides both client
and server configs. Furthermore, the properties from the *-default.xml files should never
be saved into the job.xml.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message