hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: Set variables in mapper
Date Mon, 02 Aug 2010 17:03:05 GMT
And since it is an integer you're looking for, use the utility methods
JobConf.setInt and JobConf.getInt:

Integer N = Integer.parseInt(args[2]);
JobConf.setInt("your.pack.some.name", N);

And in the Mapper's "@Override void configure(JobConf conf)", do:
conf.getInt("your.pack.some.name", 1 /* Or other default value */);

On Mon, Aug 2, 2010 at 9:53 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
> On Mon, Aug 2, 2010 at 12:17 PM, Erik Test <erik.shiken@gmail.com> wrote:
>> Hi,
>>
>> I'm trying to set a variable in my mapper class by reading an argument from
>> the command line and then passing the entry to the mapper from main. Is this
>> possible?
>>
>>  public static void main(String[] args) throws Exception
>>  {
>>    JobConf conf = new JobConf(DistanceCalc2.class);
>>    conf.setJobName("Calculate Distances");
>>
>>    conf.setOutputKeyClass(Text.class);
>>    conf.setOutputValueClass(DoubleWritable.class);
>>
>>    conf.setMapperClass(Map.class);
>>    //conf.setReducerClass(Reduce.class);
>>
>>    conf.setInputFormat(TextInputFormat.class);
>>    conf.setOutputFormat(TextOutputFormat.class);
>>
>>    FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>    FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>
>>    Map.setN(args[2]);
>>
>>    JobClient.runJob(conf);
>>  }//main
>>
>>
>>  public static class Map extends MapReduceBase
>>    implements Mapper<LongWritable, Text,
>>      Text, DoubleWritable>
>>        {
>>               ...
>>               private static int N;
>>
>>               ...
>>
>>               public void map(LongWritable key, Text value,
>>                 OutputCollector<Text, DoubleWritable> output,
>>                  Reporter reporter) throws IOException
>>                {
>>                    ....
>>                    dim = tokens.length / N;
>>                    ...
>>                }
>>
>>               public static void setN(String newN)
>>               {
>>                  N = Integer.parseInt(newN);
>>               }
>>        }
>>
>> I've tried the code above but I get an error saying that I'm dividing by
>> zero. Obviously, the argument I enter for N isn't being set as specified.
>> Erik
>>
>
> You can pass variables to the Job using the JobConf class.
>
> In your Driver class:
> jobConf.set("clone_path", clonePath);
>
> Then in your mapper / reducer override configure:
>
>  private JobConf jobConf;
>  public void configure(JobConf jobConf) {
>        super.configure(jobConf);
>        this.jobConf=jobConf;
>  }
>



-- 
Harsh J
www.harshj.com

Mime
View raw message