Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of alan.ratner@ngc.com
 designates 134.223.120.78 as permitted sender)
From: "Ratner, Alan S (IS)" <Alan.Ratner@ngc.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Subject: setInt & getInt 
Thread-Topic: setInt & getInt 
Thread-Index: AcyCsph97yl6bNdlRI6Wo52BIBfQjw==
Date: Tue, 4 Oct 2011 16:28:10 +0000
Message-ID: <C155E15D0707204885EB0BA11E54B1DF09E1B0CA@XMBVAG72.northgrum.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_C155E15D0707204885EB0BA11E54B1DF09E1B0CAXMBVAG72northgr_"
MIME-Version: 1.0

--_000_C155E15D0707204885EB0BA11E54B1DF09E1B0CAXMBVAG72northgr_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I have no problem with Hadoop.mapred using JobConf to setInt integers and p=
ass them to my map(s) for getInt as shown in the first program below.  Howe=
ver, when I use Hadoop.mapreduce using Configuration to setInt these values=
 are invisible to my map's getInt's.  Please tell me what I am doing wrong.=
  Thanks.

Both programs expect to see a file with a line or two of text in a director=
y named testIn.

Alan Ratner


This program uses JobConf and setInt/getInt and works fine.  It outputs:
number =3D 12345 (from map)


package cbTest;

import java.io.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class ConfTest extends Configured implements Tool {

            @SuppressWarnings("deprecation")
            public static class MapClass extends MapReduceBase implements M=
apper<LongWritable, Text, Text, Text> {
                        public static int number;

                        public void configure(JobConf job) {
                                    number =3D job.getInt("number",-999);
                        }

                        public void map(LongWritable key, Text t, OutputCol=
lector<Text, Text> output,
                                                Reporter reporter) throws I=
OException {
                                    System.out.println("number =3D " + numb=
er);
                        }
            }

            @SuppressWarnings("deprecation")
            public int run(String[] args) throws Exception {
                        Path InputDirectory =3D new Path("testIn");
                        Path OutputDirectory =3D new Path("testOut");
                        System.out.println(">>>> Running ConfTest Program")=
;
                        JobConf conf =3D new JobConf(getConf(), ConfTest.cl=
ass);
                        conf.setInputFormat(TextInputFormat.class);
                        conf.setOutputKeyClass(Text.class);
                        conf.setOutputValueClass(Text.class);
                        conf.setMapperClass(MapClass.class);
                        conf.setInt("number", 12345);
                        FileInputFormat.setInputPaths(conf, InputDirectory)=
;
                        FileOutputFormat.setOutputPath(conf, OutputDirector=
y);
                        FileSystem fs =3D OutputDirectory.getFileSystem(con=
f);
                        fs.delete(OutputDirectory, true); //remove output o=
f prior run
                        JobClient.runJob(conf);
                        return 0;
            }

            public static void main(String[] args) throws Exception {
                        int res =3D ToolRunner.run(new Configuration(), new=
 ConfTest(), args);
                        System.exit(res);
            }
}


This program uses Configuration and setInt/getInt.  Butt getInt in neither =
map or map:configure works.  It outputs:
>>>>> Passing integer 12345 in configuration <<<<< (from run)
map numbers: -999 -1 (from map as intMapConf and intConfConf)


package cbTest;

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class Conf2Test extends Configured implements Tool
{
            public static class MapClass extends Mapper<LongWritable, Text,=
 Text, Text>
    {
                        public int intConfConf =3D -1;

                        public void configure(Configuration job) {
                                    intConfConf =3D job.getInt("number", -2=
);
                        }

        public void map(LongWritable key, Text value, Context context) thro=
ws IOException, InterruptedException
        {
                        int intMapConf =3D context.getConfiguration().getIn=
t("number", -999);
                        System.out.println("map numbers: " + intMapConf+" "=
+intConfConf);
        }
    }

    public static void main(String[] args) throws Exception
    {
        int res =3D ToolRunner.run(new Configuration(), new Conf2Test(), ar=
gs);
        System.exit(res);

    }

            public int run(String[] arg0) throws Exception {
                        Path Input_Directory =3D new Path("testIn");
                        Path Output_Directory =3D new Path("testOut");

                        Configuration conf =3D new Configuration();
                        Job job =3D new Job(conf, Conf2Test.class.getSimple=
Name());
                        job.setJarByClass(Conf2Test.class);
                        job.setMapperClass(MapClass.class);
                        job.setMapOutputKeyClass(Text.class);
                        job.setMapOutputValueClass(Text.class);
                        TextOutputFormat.setOutputPath(job, Output_Director=
y);
                        TextInputFormat.setInputPaths(job, Input_Directory)=
;
                        FileSystem hdfs =3D FileSystem.get(conf);
                        hdfs.delete(Output_Directory, true);

                        conf.setInt("number", 12345);
                        System.out.println(">>>>> Passing integer "+conf.ge=
tInt("number", -1)+" in configuration <<<<<");

                        job.waitForCompletion(true);
                        return 0;
            }
}


Alan Ratner

--_000_C155E15D0707204885EB0BA11E54B1DF09E1B0CAXMBVAG72northgr_--