hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reza" <pdp...@eden.rutgers.edu>
Subject Help with Map Reduce
Date Sat, 18 Apr 2009 17:21:19 GMT
Hello all:

I am new to Hadoop and Map Reduce. I am writing a program to analyze some
census data.

I have a general question with MapReduce:

In the Reducer, how can I separate keys to do separate calculations based
on the key? In my case, I am trying to use if statements to separate the
keys out, but for some reason, it is not doing so.

Here is a segment of my code. My mapper seems to work, I think it is my
Reducer that is messing up:

  public static class MapClass extends MapReduceBase implements
Mapper<LongWritable, Text, Text, IntWritable> {

//    private final static IntWritable one = new IntWritable(1);
        Text sex = new Text("sex");
    public void map(LongWritable key, Text value,
                    OutputCollector<Text, IntWritable> output,
                    Reporter reporter) throws IOException {
      String line = value.toString();
      StringTokenizer itr = new StringTokenizer(line,",");

:
:
                word.set(itr.nextToken());
                IntWritable sexVal = new
IntWritable(Integer.parseInt(word.toString()));        //map gender
                output.collect(sex, sexVal);

    }//end map()method
  }//end class MapClass


  public static class Reduce extends MapReduceBase
    implements Reducer<Text, IntWritable, Text, IntWritable> {


    public void reduce(Text key, Iterator<IntWritable> values,
                       OutputCollector<Text, IntWritable> output,
                       Reporter reporter) throws IOException {

                int totalPop    = 0;        //count: total population
                int numMales    = 0;        //count: males
                int numFemales  = 0;        //count: females
                Text popOut     = new Text("The total population is: ");
                if (key.equals("sex")) {
                        while (values.hasNext()) {
                                if (values.next().get() == 0)
                                        numMales++;        //number of males
                                else
                                        numFemales++;        //number of
females
                        }//end while
                totalPop = numMales + numFemales; //count of population
                }//end if
                output.collect(popOut, new IntWritable(totalPop));

    }//end reduce method
  }//end class Reduce

key is of type Text, I also used: if (key.toString() == "sex) {....
but that wasnt working either.

There are of course other keys and values, but I would think to gather all
the keys "sex" would be done with the way I show it above.  However, My
output reads as 0.  I tried changing the numMales value to see if the if
statement is executed, but it is not. Am I doing something wrong?



Mime
View raw message