Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.212.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=FkdvS3nd7+Exx6fAsyuGjOCW/ggE6/kMhyfzIpSk7EU9MdJAFIiOBkseYaJlEyXalD
         HufJt0ghBX3ZpSoiakJyOIR98BDVdKhIj72JmAtxkQYa1kNZPPoDf0NjBlcqTmbFyn4N
         YzB59Gn6z8wdXXzQI0J9Ws6wW32uZbTKM47Dg=
MIME-Version: 1.0
In-Reply-To: <AANLkTinnOh8K9fmQob68hitZqfqiesCUzdv5JKjc8fZz@mail.gmail.com>
References: <AANLkTinnOh8K9fmQob68hitZqfqiesCUzdv5JKjc8fZz@mail.gmail.com>
Date: Mon, 2 Aug 2010 12:23:45 -0400
Message-ID: <AANLkTin2zCRytdSV8K0Y7f05Sjep=2BT6+ng8M=Czi94@mail.gmail.com>
Subject: Re: Set variables in mapper
From: Edward Capriolo <edlinuxguru@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Aug 2, 2010 at 12:17 PM, Erik Test <erik.shiken@gmail.com> wrote:
> Hi,
>
> I'm trying to set a variable in my mapper class by reading an argument fr=
om
> the command line and then passing the entry to the mapper from main. Is t=
his
> possible?
>
> =A0public static void main(String[] args) throws Exception
> =A0{
> =A0 =A0JobConf conf =3D new JobConf(DistanceCalc2.class);
> =A0 =A0conf.setJobName("Calculate Distances");
>
> =A0 =A0conf.setOutputKeyClass(Text.class);
> =A0 =A0conf.setOutputValueClass(DoubleWritable.class);
>
> =A0 =A0conf.setMapperClass(Map.class);
> =A0 =A0//conf.setReducerClass(Reduce.class);
>
> =A0 =A0conf.setInputFormat(TextInputFormat.class);
> =A0 =A0conf.setOutputFormat(TextOutputFormat.class);
>
> =A0 =A0FileInputFormat.setInputPaths(conf, new Path(args[0]));
> =A0 =A0FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>
> =A0 =A0Map.setN(args[2]);
>
> =A0 =A0JobClient.runJob(conf);
> =A0}//main
>
>
> =A0public static class Map extends MapReduceBase
> =A0 =A0implements Mapper<LongWritable, Text,
> =A0 =A0 =A0Text, DoubleWritable>
> =A0 =A0 =A0 =A0{
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 ...
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 private static int N;
>
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 ...
>
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 public void map(LongWritable key, Text value,
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 OutputCollector<Text, DoubleWritable> out=
put,
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Reporter reporter) throws IOException
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0....
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0dim =3D tokens.length / N;
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0...
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
>
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 public static void setN(String newN)
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0N =3D Integer.parseInt(newN);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0}
>
> I've tried the code above but I get an error saying that I'm dividing by
> zero. Obviously, the argument I enter for N isn't being set as specified.
> Erik
>

You can pass variables to the Job using the JobConf class.

In your Driver class:
jobConf.set("clone_path", clonePath);

Then in your mapper / reducer override configure:

  private JobConf jobConf;
  public void configure(JobConf jobConf) {
        super.configure(jobConf);
        this.jobConf=3DjobConf;
  }