hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clarence Gardner <clarence1...@gmail.com>
Subject Describing key value pairs
Date Wed, 08 Sep 2010 04:50:54 GMT
I'm writing my first m/r program, and seem to be having problems describing
the types of my key-value pairs.

I have this mapper
    public static class Map
      extends Mapper<LongWritable, Text, Text, CensusData>
and this reducer
    public static class Reduce
      extends Reducer<Text, CensusData, Text, IntWritable>

and this in my run() method
    job.setMapperClass(Map.class);
    job.setReducerClass(Reduce.class);

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

I'm assuming that the types given to setOutput...Class should be the types
output by the reducer. There doesn't seem to be a way to tell the Context
object what the types are. Unfortunately, all the examples I see on the web
output the same types from both the mapper and reducer.

I'm getting this error from my "context.write(county, cd)" statement in my
mapper:
java.io.IOException: Type mismatch in value from map: expected
org.apache.hadoop.io.IntWritable, recieved CensusData
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:850)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:541)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at CountyAgi$Map.map(CountyAgi.java:31)

I tried changing setOutputKeyClass to expect LongWritable instead of
IntWriteable, and the error changed correspondingly. So, how would I say
that the mapper outputs (Text, CensusData) and the reducer outputs (Text,
IntWritable)?

Mime
View raw message