hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kunsheng Chen <ke...@yahoo.com>
Subject Re: OK to remove conf.setCombinerClass(myReduce.class)?
Date Mon, 18 Aug 2008 21:57:03 GMT
Thanks for your reply. I need to set up different types for output inside Map and Reduce, those
methods are inside main method: (InDegreeReduce and InDegreeMap implements the interfaces)


 
       conf.setMapOutputKeyClass(Text.class);
       conf.setMapOutputValueClass(IntWritable.class);

       conf.setOutputKeyClass(Text.class);
       conf.setOutputValueClass(Text.class);

       conf.setMapperClass(InDegreeMap.class);
       
       //  I have to denote the following line otherwise the exception come out when running
      // conf.setCombinerClass(InDegreeReduce.class);  
       
       conf.setReducerClass(InDegreeReduce.class);


I have to denote 'conf.setCombinerClass(...)' otherwise it reminds me errors as below when
running (compiled successfully)

------------------------------------------

08/08/18 17:45:52 INFO mapred.JobClient: Task Id : task_200808161218_0077_m_000001_0, Status
: FAILED
java.io.IOException: wrong value class: org.apache.hadoop.io.Text is not class org.apache.hadoop.io.IntWritable
	at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:998)
	at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1083)
	at DFS_InDegree$InDegreeReduce.reduce(DFS_InDegree.java:77)
	at DFS_InDegree$InDegreeReduce.reduce(DFS_InDegree.java:52)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:876)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:782)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:694)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)

--------------------------------------------

I think there are something happen when the combiner try to handle map and reduce but don't
know what exactly it is.  thanks for your replay again. 

Please let me know if you have any idea. 

Best,

Keye

--- On Mon, 8/18/08, Ted Dunning <ted.dunning@gmail.com> wrote:

> From: Ted Dunning <ted.dunning@gmail.com>
> Subject: Re: OK to remove conf.setCombinerClass(myReduce.class)?
> To: core-dev@hadoop.apache.org, hadoopchan@yahoo.com
> Date: Monday, August 18, 2008, 8:57 PM
> This doesn't sound right.
> 
> There are two reasons.
> 
> First, the combiner is critical for good performance on
> word count
> applications.
> 
> Secondly, having a combiner should not prevent having a
> different kind of
> collector.  The combiner should just look like the reducer.
> 
> Why do you think that having the combiner is causing your
> problem?
> 
> On Mon, Aug 18, 2008 at 1:32 PM, Kunsheng Chen
> <hadoopchan@yahoo.com> wrote:
> 
> > Hello, everyone.
> >
> > I am doing  example  WordCount and have to remove
> > conf.setCombinerClass(myReduce.class) in main method
> in order to have maper
> > and reduce run with different types of output
> collector.
> >
> > I am not sure whether it cause the problem for the
> perfomance.  Please let
> > me know if it does.   Any idea is appreciated.
> >
> >
> >
> >
> 
> 
> 
> 
> -- 
> ted


      


Mime
View raw message