hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Twensky" <jim.twen...@gmail.com>
Subject Different Map and Reduce output types - weird error message
Date Fri, 29 Aug 2008 22:51:22 GMT
Hello, I am working on a Hadoop application that produces different
(key,value) types after the map and reduce phases so I'm aware that I need
to use "JobConf.setMapOutputKeyClass" and "JobConf.setMapOutputValueClass".
However, I still keep getting the following runtime error when I run my
application:

java.io.IOException: wrong value class: org.apache.hadoop.io.FloatWritable
is not class org.apache.hadoop.io.IntWritable
        at
org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:938)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$1.collect(MapTask.java:414)
        at
test.DistributionCreator$Reduce.reduce(DistributionCreator.java:104)
        at
test.DistributionCreator$Reduce.reduce(DistributionCreator.java:85)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:439)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(MapTask.java:418)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:604)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:193)
        at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1804)

My mapper class goes like:

  public static class MapClass extends MapReduceBase
    implements Mapper<LongWritable, Text, IntWritable, IntWritable> {

    (...)

    public void map(LongWritable key, Text value,
                    OutputCollector<IntWritable, IntWritable> output,
                    Reporter reporter) throws IOException {

                     (...)
    }

}

and my Reducer goes like:

  public static class Reduce extends MapReduceBase
    implements Reducer<IntWritable, IntWritable, IntWritable, FloatWritable>
{

    (...)

        public void reduce(IntWritable key, Iterator<IntWritable> values,
                       OutputCollector<IntWritable, FloatWritable> output,
                       Reporter reporter) throws IOException {


                        float sum = 0;

                        (...)

                        output.collect(key, new FloatWritable(sum));

         }

   }

   and the corresponding part of my configuration goes as follows:

    conf.setMapOutputValueClass(IntWritable.class);
    conf.setMapOutputKeyClass(IntWritable.class);
    conf.setOutputKeyClass(IntWritable.class);
    conf.setOutputValueClass(FloatWritable.class);

   which I believe is consistent with the mapper and the reducer classes.
Can you please let me know what I'm missing here?

   Thanks in advance,

   Jim

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message