hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Costa <psdc1...@gmail.com>
Subject Re: PiEstimator error - Type mismatch in key from map
Date Thu, 27 Jan 2011 10:36:24 GMT
The map output class are well defined:
keyClass: class org.apache.hadoop.io.BooleanWritable - valClass: class
org.apache.hadoop.io.LongWritable

but executing the pi example, the values that map function passes is:
keyClass: class org.apache.hadoop.io.LongWritable - valClass: class
org.apache.hadoop.io.Text


I looked at the PiEstimator.class.PiMapper#map function, and the
output collector seems ok.

[code]
public void map(LongWritable offset,
		LongWritable size,
		OutputCollector<BooleanWritable, LongWritable> out,
		Reporter reporter) throws IOException {
(...)
out.collect(new BooleanWritable(true), new LongWritable(numInside));
out.collect(new BooleanWritable(false), new LongWritable(numOutside));
}
[/code]

I'm really confused, right now. How can this be happening?


On Thu, Jan 27, 2011 at 10:19 AM, Pedro Costa <psdc1978@gmail.com> wrote:
> Thanks Nicholas, but it didn't worked.
>
> Can I do a remote debugging on hadoop examples? I really like to put a
> breakpoint in the Pi class.
>
> Thanks,
>
> On Wed, Jan 26, 2011 at 6:46 PM, Tsz Wo (Nicholas), Sze
> <s29752-hadoopuser@yahoo.com> wrote:
>> Okay, I got it now.  You were talking about your programs but not the
>> PiEstimator example came from Hadoop.  Then, you have to set
>> "mapred.output.key.class" and "mapred.output.value.class" as Srihari
>> mentioned.  Below are the APIs.
>>
>>     //new API
>>     final Job job = ...
>>     job.setMapOutputKeyClass(BooleanWritable.class);
>>     job.setMapOutputValueClass(LongWritable.class);
>>
>>     //old API
>>     final JobConf jobconf = ...
>>     jobconf.setOutputKeyClass(BooleanWritable.class);
>>     jobconf.setOutputValueClass(LongWritable.class);
>>
>> Nicholas
>>
>> ________________________________
>> From: Srihari Anantha Padmanabhan <sriharia@yahoo-inc.com>
>> To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
>> Sent: Wed, January 26, 2011 10:36:09 AM
>> Subject: Re: PiEstimator error - Type mismatch in key from map
>>
>> I am using Hadoop 0.20.2. I just wrote my own map-reduce program based on
>> the map-reduce tutorial at
>> http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html
>>
>> On Jan 26, 2011, at 10:27 AM, Pedro Costa wrote:
>>
>>> Hadoop 20.1
>>>
>>> On Wed, Jan 26, 2011 at 6:26 PM, Tsz Wo (Nicholas), Sze
>>> <s29752-hadoopuser@yahoo.com> wrote:
>>>> Hi Srihari,
>>>>
>>>> Same questions to you: Which version of Hadoop are you using?  And where
>>>> did
>>>> you get the examples?  I guess you were able to reproduce it.  I suspect
>>>> the
>>>> examples and the Hadoop are in different versions.
>>>>
>>>> Nicholas
>>>>
>>>>
>>>> ________________________________
>>>> From: Srihari Anantha Padmanabhan <sriharia@yahoo-inc.com>
>>>> To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
>>>> Sent: Wed, January 26, 2011 10:15:08 AM
>>>> Subject: Re: PiEstimator error - Type mismatch in key from map
>>>>
>>>> I got a similar error before in one of my projects. I had to set the
>>>> values
>>>> for "mapred.output.key.class" and "mapred.output.value.class".
>>>> That resolved the issue for me.
>>>> Srihari
>>>> On Jan 26, 2011, at 10:09 AM, Pedro Costa wrote:
>>>>
>>>> Yes, I can reproduce it deterministically. But, I also did some
>>>> changes to the Hadoop MR code. Most definitely this is the reason. I'm
>>>> looking throughly through the code.
>>>>
>>>> I'll say something after I find the problem.
>>>>
>>>> I was just wondering if this error has happened to someone before.
>>>> Maybe I could get a hint and try to see what's my problem easily.
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, Jan 26, 2011 at 6:02 PM, Tsz Wo (Nicholas), Sze
>>>> <s29752-hadoopuser@yahoo.com> wrote:
>>>>
>>>> Hi Pedro,
>>>>
>>>> This is interesting.  Which version of Hadoop are you using?  And where
>>>> did
>>>>
>>>> you get the example class files?  Also, are you able to reproduce it
>>>>
>>>> deterministically?
>>>>
>>>> Nicholas
>>>>
>>>> ________________________________
>>>>
>>>> From: Pedro Costa <psdc1978@gmail.com>
>>>>
>>>> To: mapreduce-user@hadoop.apache.org
>>>>
>>>> Sent: Wed, January 26, 2011 5:47:01 AM
>>>>
>>>> Subject: PiEstimator error - Type mismatch in key from map
>>>>
>>>> Hi,
>>>>
>>>> I run the PI example of hadoop, and I've got the following error:
>>>>
>>>> [code]
>>>>
>>>> java.io.IOException: Type mismatch in key from map: expected
>>>>
>>>> org.apache.hadoop.io.BooleanWritable, recieved
>>>>
>>>> org.apache.hadoop.io.LongWritable
>>>>
>>>>    at
>>>>
>>>>
>>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:885)
>>>>
>>>>    at
>>>>
>>>>
>>>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:551)
>>>>
>>>>    at
>>>>
>>>>
>>>> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:81)
>>>>
>>>>    at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
>>>>
>>>>    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>
>>>>    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:637)
>>>>
>>>>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>>>>
>>>>    at org.apache.hadoop.mapred.Child.main(Child.java:190)
>>>>
>>>> [/code]
>>>>
>>>> I've look at the map function of the class "PiEstimator.class" and it
>>>> seems
>>>>
>>>> ok.
>>>>
>>>> [code]
>>>>
>>>> public void map(LongWritable offset,
>>>>
>>>>        LongWritable size,
>>>>
>>>>        OutputCollector<BooleanWritable, LongWritable> out,
>>>>
>>>>        Reporter reporter) throws IOException {}
>>>>
>>>> [/code]
>>>>
>>>>
>>>> What's wrong with this examples?
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>>
>>>> Pedro
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Pedro
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Pedro
>>
>>
>
>
>
> --
> Pedro
>



-- 
Pedro

Mime
View raw message