hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Kozlov <ale...@cloudera.com>
Subject Re: Problem with Reducer emitting a different Key than Mapper
Date Wed, 16 Jun 2010 16:26:11 GMT
Hi Steve, did you do

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);

?

Alex K

On Wed, Jun 16, 2010 at 9:15 AM, Steve Lewis <lordjoe2000@gmail.com> wrote:

> Problem with Reducer emitting a different Key than Mapper
> IO have the FOllowing code where the Mapper emits a custom Key and the
> reducer isa expected to emit text
>
> Using Hadoop 0.2 on a local instance  I asj the reducer to write Text,Text
> - thiew is even what the IDE says I should do
> and what I get is the exception below - any bright ideas???
>
> public class PartitionReducer extends Reducer<GenonePartitionKey, Text,
> Text, Text>
> {
>
>   /**
>      * This method is called once for each key. Most applications will
> define
>      * their reduce class by overriding this method. The default
> implementation
>      * is an identity function.
>      */
>     @Override
>     protected void reduce(GenonePartitionKey key, Iterable<Text> values,
> Context context)
>             throws IOException, InterruptedException
>     {
>           context.write(new Text("Foo"),new Text("Bar"));
>
>      ....
>  }
>
>  }
>
>
> 10/06/16 09:08:45 INFO mapred.MapTask: Starting flush of map output
> 10/06/16 09:08:45 WARN mapred.LocalJobRunner: job_local_0001
> java.io.IOException: Spill failed
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1123)
>  at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> Caused by: java.io.IOException: wrong key class: class
> org.apache.hadoop.io.Text is not class
> org.systemsbiology.hadoop.GenonePartitionKey
> at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:164)
>  at
> org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:880)
> at
> org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1201)
>  at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at
> org.systemsbiology.hadoop.PartitionReducer.reduce(PartitionReducer.java:165)
>  at
> org.systemsbiology.hadoop.PartitionReducer.reduce(PartitionReducer.java:23)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>  at
> org.systemsbiology.hadoop.PartitionReducer.run(PartitionReducer.java:259)
> at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1222)
>  at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1265)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:686)
>  at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1173)
>
>
> --
> Steven M. Lewis PhD
> Institute for Systems Biology
> Seattle WA
>

Mime
View raw message