hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venu Gopala Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7239) The variables in the configuration files are not replaced with values, when the job is submitted by Job client
Date Fri, 22 Apr 2011 18:13:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023313#comment-13023313

Venu Gopala Rao commented on HADOOP-7239:

I looked at code for the root cause. It is as follows

While writing the job.xml, it is using the Configuration.writeXML(), which is as follows

Properties properties = getProps();
    try {

      for (Enumeration e = properties.keys(); e.hasMoreElements();) {
        String name = (String)e.nextElement();
        Object object = properties.get(name);
        String value = null;
        if (object instanceof String) {
          value = (String) object;
        }else {

So here all properties in configuration are retrieved using getProps() which returns a ConcurrentHashMap
and written to XML from that. So the variables are not substituted. instead of using *properties.get(name);*
in above snippet we should use *get(name)*, so that all variable substitution happen correctly.

> The variables in the configuration files are not replaced with values, when the job is
submitted by Job client
> --------------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-7239
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7239
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.21.0
>         Environment: Linux + Sun JDK 1.6
>            Reporter: Venu Gopala Rao
> We have a case where we wanted to create the Job names dynamically at run time.Since
JobConf is an extension for the Configuration object, we thought we can make use of the variable
substitution concept in configuration like below
> Job job = new Job(conf, "${mapred.user.name}" + "-job" + new Random().nextInt()); 
> job.setJarByClass(WordCount.class); 
> job.setMapperClass(TokenizerMapper.class); 
> job.setCombinerClass(IntSumReducer.class); 
> job.setReducerClass(IntSumReducer.class); 
> job.setOutputKeyClass(Text.class); 
> job.setOutputValueClass(IntWritable.class); 
> FileInputFormat.addInputPath(job, new Path(otherArgs[0])); 
> FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); 
> job.submit(); 
> We set the required run time variables(in this case mapred.user.name) before calling
the job submit. But on the Job tracker side the variables are not replaced correctly.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message