hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5047) Hod should be modified to generate core-site.xml, mapred-site.xml and hdfs-site.xml
Date Tue, 27 Jan 2009 14:52:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667688#action_12667688
] 

Hemanth Yamijala commented on HADOOP-5047:
------------------------------------------

HOD generates hadoop-site.xml on the client, and on the cluster nodes where it brings up the
daemons. There are two types of configuration items it writes into the hadoop-site.xml:

- Entries it generates values for. E.g. the jobtracker URL
- Entries it is given via the hod configuration file. These are user specified entries.

It is easy to split the first category. Since the list is a hardcoded set of configuration
items, we can hardcode the decision of which file (core, mapred, or dfs) this item belongs
to and generate the entries into the appropriate file.

For the user specified entries, the decision cannot be taken by HOD in a maintainable fashion.
Further, it causes more of a problem because of interface and usability issues.

Currently, HOD has a very convenient command line syntax to specify the list of configuration
items, as follows:

-Cname=value - client params
-Mname=value - Mapred server params
-Fname=value - Final Mapred server params
-Hname=value - HDFS server params
-Sname=value - Final HDFS server params.

One approach to splitting the configuration is to split each of the categories into 3 as core,
mapred and hdfs. However, I feel this might be too cumbersome for users to express.

So, a proposal is to introduce this syntax:
{code}
-<Option Letter>[category:]<name>=<value>, where category is 'core' or 'mapred'
or 'hdfs'.
{code}

For e.g.
{code}
-Mmapred:mapred.reduce.tasks=4 or -Ccore:dfs.client.buffer.size=1000000
{code}

Based on the category, the configuration item can be generated to the appropriate configuration
file.

For backwards compatibility, we can say that if category is not specified, the configuration
item would be generated in the hadoop-site.xml file. When hadoop-site.xml is completely removed
from hadoop, HOD can also stop supporting options where the category is not specified.

Comments ?

> Hod should be modified to generate core-site.xml, mapred-site.xml and hdfs-site.xml
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-5047
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5047
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/hod
>    Affects Versions: 0.20.0
>            Reporter: Suman Sehgal
>            Assignee: Peeyush Bishnoi
>
> Hod is currently generating hadoop-site.xml but as per the modifications done in hadoop
(ref: JIRA-4631), hod should also be modified to be in synch with this change. It should also
generate core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml from
0.20.0 onwards.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message