Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <23610684.1186082033010.JavaMail.jira@brutus>
Date: Thu, 2 Aug 2007 12:13:53 -0700 (PDT)
From: "Michael Bieniosek (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-785) Divide the server and client
 configurations
In-Reply-To: <4048066.1165393281134.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517343 ] 

Michael Bieniosek commented on HADOOP-785:
------------------------------------------

Arun,

Your proposal sounds reasonable.  Thanks for looking at this issue.

Currently, hadoop-default.xml is not supposed to be changed by users.  Would you relax this convention in your proposal?  There might be a few variables that I'd like to set for client and server at the same time (eg. namenode address).

Why don't you want to split up namenode vs. jobtracker and datanode vs. tasktracker?  I understand that it's desirable to keep things simple, but dfs and mapreduce don't interact very much in terms of their configs, so there is a natural separation.

Instead of dividing configs into "beginner" and "advanced", we should think about dividing into "things you probably need to change" (at the top of the file) and "things you probably don't need to change" (at the bottom of the file).  This division could be done with xml comments -- I don't think it needs to be so formal as to need a new field.  


> Divide the server and client configurations
> -------------------------------------------
>
>                 Key: HADOOP-785
>                 URL: https://issues.apache.org/jira/browse/HADOOP-785
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.9.0
>            Reporter: Owen O'Malley
>            Assignee: Arun C Murthy
>             Fix For: 0.15.0
>
>
> The configuration system is easy to misconfigure and I think we need to strongly divide the server from client configs. 
> An example of the problem was a configuration where the task tracker has a hadoop-site.xml that set mapred.reduce.tasks to 1. Therefore, the job tracker had the right number of reduces, but the map task thought there was a single reduce. This lead to a hard to find diagnose failure.
> Therefore, I propose separating out the configuration types as:
> class Configuration;
> // reads site-default.xml, hadoop-default.xml
> class ServerConf extends Configuration;
> // reads hadoop-server.xml, $super
> class DfsServerConf extends ServerConf;
> // reads dfs-server.xml, $super
> class MapRedServerConf extends ServerConf;
> // reads mapred-server.xml, $super
> class ClientConf extends Configuration;
> // reads hadoop-client.xml, $super
> class JobConf extends ClientConf;
> // reads job.xml, $super
> Note in particular, that nothing corresponds to hadoop-site.xml, which overrides both client and server configs. Furthermore, the properties from the *-default.xml files should never be saved into the job.xml.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.