hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Arenas <eare...@rocketmail.com>
Subject Re: Does the class of the Mapper output need to match the exact class of the specified output?
Date Wed, 27 Jan 2010 17:27:56 GMT
Hi Chris,

what you need to do is

(with Hadoop 0.20+)

job.setMapOutputValueClass(Text.class); //Mapper
job.setMapOutputKeyClass(LongWritable .class); //Mapper

This will do the trick.

regards,
Eric Arenas




________________________________
From: "Wilkes, Chris" <cwilkes@gmail.com>
To: mapreduce-user@hadoop.apache.org
Sent: Tue, January 26, 2010 5:17:06 PM
Subject: Does the class of the Mapper output need to match the exact class of the specified
output?

I'm outputting a Text and a LongWritable in my mapper and told the job that my mapper output
class is Writable (the interface shared by both of them):
  job.setMapOutputValueClass(Writable.class);
I'm doing this as I have two different types of input files and am combining them together.
 I could write them both as as Text but then I'll have to put a marker in front of the tag
to indicate what type of entry it is instead of doing a
  if (value instanceof Text) { }  else if (value instanceof LongWritable) { }


This exception is thrown:

java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.Writable,
recieved org.apache.hadoop.io.LongWritable
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:812)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)

The MapTask code (which is being used even though I'm using the new API) shows that a != is
used to compare the classes:
http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/MapTask.java?view=log

 if (value.getClass() != valClass) {
        throw new IOException("Type mismatch in value from map: expected "
                              + valClass.getName() + ", recieved "
                              + value.getClass().getName());
      }

Does this level of checking really need to be done?  Could it just be a Class.isAssignableFrom()
check?

Mime
View raw message