hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johannes Zillmann ...@101tec.com>
Subject Re: Different output classes from map and reducer
Date Thu, 28 Feb 2008 17:51:30 GMT
Hi Slitz,

try
conf.setMapOutputValueClass(Text.class);
conf.setMapOutputKeyClass(Text.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);

Johannes

slitz wrote:
> Hello,
> I'm experimenting with hadoop a few days now, but i'm stuck trying to output
> different classes from map and reduce methods.
>
> I have something like:
>
> class test {
>
> public static class Map extends MapReduceBase implements
> Mapper<LongWritable, Text, Text, Text> {
>        public void map(LongWritable key, Text value, OutputCollector<Text,
> Text> output, Reporter reporter) throws IOException {
>
>        (...)
>
>        output.collect(new Text(...), new Text(...));
>        }
> }
>
> public static class Reduce extends MapReduceBase implements Reducer<Text,
> Text, Text, IntWritable> {
>         public void reduce(Text key, Iterator<Text> values,
> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
> IOException {
>
>
>          (...)
>
>           output.collect(key, new IntWritable(...));
>
>          }
>
> }
>
> }
>
>
> the relevant part of my conf goes like:
>
> JobConf conf = new JobConf(test.class);
> conf.setOutputKeyClass(Text.class);
> conf.setOutputValueClass(Text.class);
>
> conf.setInputFormat(TextInputFormat.class);
> conf.setOutputFormat(TextOutputFormat.class);
>
> i keep getting this error:
>
> 08/02/28 01:52:47 INFO mapred.JobClient:  map 50% reduce 0%
> 08/02/28 01:52:51 INFO mapred.JobClient: Task Id :
> task_200802261545_0032_m_000001_0, Status : FAILED
> java.io.IOException: wrong value class: org.apache.hadoop.io.IntWritable is
> not class org.apache.hadoop.io.Text
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:938)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$1.collect(MapTask.java
> :414)
> at org.myorg.WordCount$Reduce.reduce(WordCount.java:64)
> at org.myorg.WordCount$Reduce.reduce(WordCount.java:49)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(
> MapTask.java:439)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(
> MapTask.java:418)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:604)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:193)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1804)
>
>
> if in the conf i switch this -> conf.setOutputValueClass(Text.class); (map's
> output value  type
> for this -> conf.setOutputValueClass(IntWritable.class); (reduce's output
> value type)
>
> then i get this:
>
> 08/02/28 02:05:08 INFO mapred.JobClient:  map 50% reduce 0%
> 08/02/28 02:05:12 INFO mapred.JobClient: Task Id :
> task_200802261545_0033_m_000001_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.Text
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java
> :336)
> at org.myorg.WordCount$Map.map(WordCount.java:43)
> at org.myorg.WordCount$Map.map(WordCount.java:16)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1804)
>
>
> i'm just trying to modify slightly the wordcount example to fit my needs but
> i keep getting this kind of errors.
> Can somebody please point me the right direction?
>
>
> Thank you
>
> slitz
>
>   


-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
101tec GmbH

Halle (Saale), Saxony-Anhalt, Germany
http://www.101tec.com


Mime
View raw message