hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michel Tourn (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-345) JobConf access to name-values
Date Thu, 06 Jul 2006 00:42:29 GMT
JobConf access to name-values

         Key: HADOOP-345
         URL: http://issues.apache.org/jira/browse/HADOOP-345
     Project: Hadoop
        Type: Improvement

    Reporter: Michel Tourn
 Assigned to: Michel Tourn 

class JobConf (or its base class Configuration) 
should be extended to enable enumeration of all its key-value pairs.
( more precisely: the Properties returned by Configuration.getProps() )

This will be useful to "export" all JobConf properties to environment variables.
We use env.vars to expose some Hadoop context to non-Java MapReduce applications.

Note that the typed properties are also represented as Strings 
(getInt, getStrings, getClass, etc.)
So a single enumeration exposes everything as (untyped) environment variables.

The proposed escaping rules from JobConf properties to env.var are:

1. values are left as-is.
2. keys are escaped as follows:
[A-Za-z0-9] --> unchanged.
all other chars --> underscore.

For example
set("mapred.input.key.class", "com.example.MyKey")
becomes env.var:
export mapred_input_key_class=com.example.MyKey

1. Environment variables are case-sensitive. (Although uppercase is the preferred convention)
  So no need to uppercase everything.
2. Some characters are forbidden in env.vars, or at least not shell-friendly:
For example period, colon are problematic.
3. The Hadoop conventions are already hierarchical and provide some namespace protection.
   This means we don't need an additional prefix as protection.
   For example all exported environment variables will start with "mapred." , "dfs." , "ipc."
  This means they will not conflict with standard environemnt variables like PATH, USER, etc.
  And they will not conflict with standard hadoop env.vars because those are upper-case. (like

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message