hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chase Bradford <chase.bradf...@gmail.com>
Subject Re: PiEstimator error - Type mismatch in key from map
Date Thu, 27 Jan 2011 15:40:14 GMT
Are you sure the function signature for you Mapper's map matches the super class, and that
you specified your Map class in the job setup?  It sounds a bit like the base o.a.h.mapreduce.Mapper
map implementation is being used instead.


On Jan 27, 2011, at 2:36 AM, Pedro Costa <psdc1978@gmail.com> wrote:

> The map output class are well defined:
> keyClass: class org.apache.hadoop.io.BooleanWritable - valClass: class
> org.apache.hadoop.io.LongWritable
> 
> but executing the pi example, the values that map function passes is:
> keyClass: class org.apache.hadoop.io.LongWritable - valClass: class
> org.apache.hadoop.io.Text
> 
> 
> I looked at the PiEstimator.class.PiMapper#map function, and the
> output collector seems ok.
> 
> [code]
> public void map(LongWritable offset,
>        LongWritable size,
>        OutputCollector<BooleanWritable, LongWritable> out,
>        Reporter reporter) throws IOException {
> (...)
> out.collect(new BooleanWritable(true), new LongWritable(numInside));
> out.collect(new BooleanWritable(false), new LongWritable(numOutside));
> }
> [/code]
> 
> I'm really confused, right now. How can this be happening?
> 
> 
> On Thu, Jan 27, 2011 at 10:19 AM, Pedro Costa <psdc1978@gmail.com> wrote:
>> Thanks Nicholas, but it didn't worked.
>> 
>> Can I do a remote debugging on hadoop examples? I really like to put a
>> breakpoint in the Pi class.
>> 
>> Thanks,
>> 
>> On Wed, Jan 26, 2011 at 6:46 PM, Tsz Wo (Nicholas), Sze
>> <s29752-hadoopuser@yahoo.com> wrote:
>>> Okay, I got it now.  You were talking about your programs but not the
>>> PiEstimator example came from Hadoop.  Then, you have to set
>>> "mapred.output.key.class" and "mapred.output.value.class" as Srihari
>>> mentioned.  Below are the APIs.
>>> 
>>>     //new API
>>>     final Job job = ...
>>>     job.setMapOutputKeyClass(BooleanWritable.class);
>>>     job.setMapOutputValueClass(LongWritable.class);
>>> 
>>>     //old API
>>>     final JobConf jobconf = ...
>>>     jobconf.setOutputKeyClass(BooleanWritable.class);
>>>     jobconf.setOutputValueClass(LongWritable.class);
>>> 
>>> Nicholas
>>> 
>>> ________________________________
>>> From: Srihari Anantha Padmanabhan <sriharia@yahoo-inc.com>
>>> To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
>>> Sent: Wed, January 26, 2011 10:36:09 AM
>>> Subject: Re: PiEstimator error - Type mismatch in key from map
>>> 
>>> I am using Hadoop 0.20.2. I just wrote my own map-reduce program based on
>>> the map-reduce tutorial at
>>> http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html
>>> 
>>> On Jan 26, 2011, at 10:27 AM, Pedro Costa wrote:
>>> 
>>>> Hadoop 20.1
>>>> 
>>>> On Wed, Jan 26, 2011 at 6:26 PM, Tsz Wo (Nicholas), Sze
>>>> <s29752-hadoopuser@yahoo.com> wrote:
>>>>> Hi Srihari,
>>>>> 
>>>>> Same questions to you: Which version of Hadoop are you using?  And where
>>>>> did
>>>>> you get the examples?  I guess you were able to reproduce it.  I suspect
>>>>> the
>>>>> examples and the Hadoop are in different versions.
>>>>> 
>>>>> Nicholas
>>>>> 
>>>>> 
>>>>> ________________________________
>>>>> From: Srihari Anantha Padmanabhan <sriharia@yahoo-inc.com>
>>>>> To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
>>>>> Sent: Wed, January 26, 2011 10:15:08 AM
>>>>> Subject: Re: PiEstimator error - Type mismatch in key from map
>>>>> 
>>>>> I got a similar error before in one of my projects. I had to set the
>>>>> values
>>>>> for "mapred.output.key.class" and "mapred.output.value.class".
>>>>> That resolved the issue for me.
>>>>> Srihari
>>>>> On Jan 26, 2011, at 10:09 AM, Pedro Costa wrote:
>>>>> 
>>>>> Yes, I can reproduce it deterministically. But, I also did some
>>>>> changes to the Hadoop MR code. Most definitely this is the reason. I'm
>>>>> looking throughly through the code.
>>>>> 
>>>>> I'll say something after I find the problem.
>>>>> 
>>>>> I was just wondering if this error has happened to someone before.
>>>>> Maybe I could get a hint and try to see what's my problem easily.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> On Wed, Jan 26, 2011 at 6:02 PM, Tsz Wo (Nicholas), Sze
>>>>> <s29752-hadoopuser@yahoo.com> wrote:
>>>>> 
>>>>> Hi Pedro,
>>>>> 
>>>>> This is interesting.  Which version of Hadoop are you using?  And where
>>>>> did
>>>>> 
>>>>> you get the example class files?  Also, are you able to reproduce it
>>>>> 
>>>>> deterministically?
>>>>> 
>>>>> Nicholas
>>>>> 
>>>>> ________________________________
>>>>> 
>>>>> From: Pedro Costa <psdc1978@gmail.com>
>>>>> 
>>>>> To: mapreduce-user@hadoop.apache.org
>>>>> 
>>>>> Sent: Wed, January 26, 2011 5:47:01 AM
>>>>> 
>>>>> Subject: PiEstimator error - Type mismatch in key from map
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I run the PI example of hadoop, and I've got the following error:
>>>>> 
>>>>> [code]
>>>>> 
>>>>> java.io.IOException: Type mismatch in key from map: expected
>>>>> 
>>>>> org.apache.hadoop.io.BooleanWritable, recieved
>>>>> 
>>>>> org.apache.hadoop.io.LongWritable
>>>>> 
>>>>>     at
>>>>> 
>>>>> 
>>>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:885)
>>>>> 
>>>>>     at
>>>>> 
>>>>> 
>>>>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:551)
>>>>> 
>>>>>     at
>>>>> 
>>>>> 
>>>>> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:81)
>>>>> 
>>>>>     at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
>>>>> 
>>>>>     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>> 
>>>>>     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:637)
>>>>> 
>>>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>>>>> 
>>>>>     at org.apache.hadoop.mapred.Child.main(Child.java:190)
>>>>> 
>>>>> [/code]
>>>>> 
>>>>> I've look at the map function of the class "PiEstimator.class" and it
>>>>> seems
>>>>> 
>>>>> ok.
>>>>> 
>>>>> [code]
>>>>> 
>>>>> public void map(LongWritable offset,
>>>>> 
>>>>>         LongWritable size,
>>>>> 
>>>>>         OutputCollector<BooleanWritable, LongWritable> out,
>>>>> 
>>>>>         Reporter reporter) throws IOException {}
>>>>> 
>>>>> [/code]
>>>>> 
>>>>> 
>>>>> What's wrong with this examples?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> --
>>>>> 
>>>>> Pedro
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Pedro
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Pedro
>>> 
>>> 
>> 
>> 
>> 
>> --
>> Pedro
>> 
> 
> 
> 
> -- 
> Pedro

Mime
View raw message