hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Set variables in mapper
Date Mon, 02 Aug 2010 16:23:45 GMT
On Mon, Aug 2, 2010 at 12:17 PM, Erik Test <erik.shiken@gmail.com> wrote:
> Hi,
>
> I'm trying to set a variable in my mapper class by reading an argument from
> the command line and then passing the entry to the mapper from main. Is this
> possible?
>
>  public static void main(String[] args) throws Exception
>  {
>    JobConf conf = new JobConf(DistanceCalc2.class);
>    conf.setJobName("Calculate Distances");
>
>    conf.setOutputKeyClass(Text.class);
>    conf.setOutputValueClass(DoubleWritable.class);
>
>    conf.setMapperClass(Map.class);
>    //conf.setReducerClass(Reduce.class);
>
>    conf.setInputFormat(TextInputFormat.class);
>    conf.setOutputFormat(TextOutputFormat.class);
>
>    FileInputFormat.setInputPaths(conf, new Path(args[0]));
>    FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>
>    Map.setN(args[2]);
>
>    JobClient.runJob(conf);
>  }//main
>
>
>  public static class Map extends MapReduceBase
>    implements Mapper<LongWritable, Text,
>      Text, DoubleWritable>
>        {
>               ...
>               private static int N;
>
>               ...
>
>               public void map(LongWritable key, Text value,
>                 OutputCollector<Text, DoubleWritable> output,
>                  Reporter reporter) throws IOException
>                {
>                    ....
>                    dim = tokens.length / N;
>                    ...
>                }
>
>               public static void setN(String newN)
>               {
>                  N = Integer.parseInt(newN);
>               }
>        }
>
> I've tried the code above but I get an error saying that I'm dividing by
> zero. Obviously, the argument I enter for N isn't being set as specified.
> Erik
>

You can pass variables to the Job using the JobConf class.

In your Driver class:
jobConf.set("clone_path", clonePath);

Then in your mapper / reducer override configure:

  private JobConf jobConf;
  public void configure(JobConf jobConf) {
        super.configure(jobConf);
        this.jobConf=jobConf;
  }

Mime
View raw message