hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Costa <psdc1...@gmail.com>
Subject Re: PiEstimator error - Type mismatch in key from map
Date Thu, 27 Jan 2011 16:29:15 GMT
Yes, that's the one that's being used ( o.a.h.mapreduce.Mapper ). This
is not the right one to use?



On Thu, Jan 27, 2011 at 3:40 PM, Chase Bradford
<chase.bradford@gmail.com> wrote:
> Are you sure the function signature for you Mapper's map matches the super class, and
that you specified your Map class in the job setup?  It sounds a bit like the base o.a.h.mapreduce.Mapper
map implementation is being used instead.
>
>
> On Jan 27, 2011, at 2:36 AM, Pedro Costa <psdc1978@gmail.com> wrote:
>
>> The map output class are well defined:
>> keyClass: class org.apache.hadoop.io.BooleanWritable - valClass: class
>> org.apache.hadoop.io.LongWritable
>>
>> but executing the pi example, the values that map function passes is:
>> keyClass: class org.apache.hadoop.io.LongWritable - valClass: class
>> org.apache.hadoop.io.Text
>>
>>
>> I looked at the PiEstimator.class.PiMapper#map function, and the
>> output collector seems ok.
>>
>> [code]
>> public void map(LongWritable offset,
>>        LongWritable size,
>>        OutputCollector<BooleanWritable, LongWritable> out,
>>        Reporter reporter) throws IOException {
>> (...)
>> out.collect(new BooleanWritable(true), new LongWritable(numInside));
>> out.collect(new BooleanWritable(false), new LongWritable(numOutside));
>> }
>> [/code]
>>
>> I'm really confused, right now. How can this be happening?
>>
>>
>> On Thu, Jan 27, 2011 at 10:19 AM, Pedro Costa <psdc1978@gmail.com> wrote:
>>> Thanks Nicholas, but it didn't worked.
>>>
>>> Can I do a remote debugging on hadoop examples? I really like to put a
>>> breakpoint in the Pi class.
>>>
>>> Thanks,
>>>
>>> On Wed, Jan 26, 2011 at 6:46 PM, Tsz Wo (Nicholas), Sze
>>> <s29752-hadoopuser@yahoo.com> wrote:
>>>> Okay, I got it now.  You were talking about your programs but not the
>>>> PiEstimator example came from Hadoop.  Then, you have to set
>>>> "mapred.output.key.class" and "mapred.output.value.class" as Srihari
>>>> mentioned.  Below are the APIs.
>>>>
>>>>     //new API
>>>>     final Job job = ...
>>>>     job.setMapOutputKeyClass(BooleanWritable.class);
>>>>     job.setMapOutputValueClass(LongWritable.class);
>>>>
>>>>     //old API
>>>>     final JobConf jobconf = ...
>>>>     jobconf.setOutputKeyClass(BooleanWritable.class);
>>>>     jobconf.setOutputValueClass(LongWritable.class);
>>>>
>>>> Nicholas
>>>>
>>>> ________________________________
>>>> From: Srihari Anantha Padmanabhan <sriharia@yahoo-inc.com>
>>>> To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
>>>> Sent: Wed, January 26, 2011 10:36:09 AM
>>>> Subject: Re: PiEstimator error - Type mismatch in key from map
>>>>
>>>> I am using Hadoop 0.20.2. I just wrote my own map-reduce program based on
>>>> the map-reduce tutorial at
>>>> http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html
>>>>
>>>> On Jan 26, 2011, at 10:27 AM, Pedro Costa wrote:
>>>>
>>>>> Hadoop 20.1
>>>>>
>>>>> On Wed, Jan 26, 2011 at 6:26 PM, Tsz Wo (Nicholas), Sze
>>>>> <s29752-hadoopuser@yahoo.com> wrote:
>>>>>> Hi Srihari,
>>>>>>
>>>>>> Same questions to you: Which version of Hadoop are you using?  And
where
>>>>>> did
>>>>>> you get the examples?  I guess you were able to reproduce it.  I
suspect
>>>>>> the
>>>>>> examples and the Hadoop are in different versions.
>>>>>>
>>>>>> Nicholas
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Srihari Anantha Padmanabhan <sriharia@yahoo-inc.com>
>>>>>> To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
>>>>>> Sent: Wed, January 26, 2011 10:15:08 AM
>>>>>> Subject: Re: PiEstimator error - Type mismatch in key from map
>>>>>>
>>>>>> I got a similar error before in one of my projects. I had to set
the
>>>>>> values
>>>>>> for "mapred.output.key.class" and "mapred.output.value.class".
>>>>>> That resolved the issue for me.
>>>>>> Srihari
>>>>>> On Jan 26, 2011, at 10:09 AM, Pedro Costa wrote:
>>>>>>
>>>>>> Yes, I can reproduce it deterministically. But, I also did some
>>>>>> changes to the Hadoop MR code. Most definitely this is the reason.
I'm
>>>>>> looking throughly through the code.
>>>>>>
>>>>>> I'll say something after I find the problem.
>>>>>>
>>>>>> I was just wondering if this error has happened to someone before.
>>>>>> Maybe I could get a hint and try to see what's my problem easily.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> On Wed, Jan 26, 2011 at 6:02 PM, Tsz Wo (Nicholas), Sze
>>>>>> <s29752-hadoopuser@yahoo.com> wrote:
>>>>>>
>>>>>> Hi Pedro,
>>>>>>
>>>>>> This is interesting.  Which version of Hadoop are you using?  And
where
>>>>>> did
>>>>>>
>>>>>> you get the example class files?  Also, are you able to reproduce
it
>>>>>>
>>>>>> deterministically?
>>>>>>
>>>>>> Nicholas
>>>>>>
>>>>>> ________________________________
>>>>>>
>>>>>> From: Pedro Costa <psdc1978@gmail.com>
>>>>>>
>>>>>> To: mapreduce-user@hadoop.apache.org
>>>>>>
>>>>>> Sent: Wed, January 26, 2011 5:47:01 AM
>>>>>>
>>>>>> Subject: PiEstimator error - Type mismatch in key from map
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I run the PI example of hadoop, and I've got the following error:
>>>>>>
>>>>>> [code]
>>>>>>
>>>>>> java.io.IOException: Type mismatch in key from map: expected
>>>>>>
>>>>>> org.apache.hadoop.io.BooleanWritable, recieved
>>>>>>
>>>>>> org.apache.hadoop.io.LongWritable
>>>>>>
>>>>>>     at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:885)
>>>>>>
>>>>>>     at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:551)
>>>>>>
>>>>>>     at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:81)
>>>>>>
>>>>>>     at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
>>>>>>
>>>>>>     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>>>
>>>>>>     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:637)
>>>>>>
>>>>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>>>>>>
>>>>>>     at org.apache.hadoop.mapred.Child.main(Child.java:190)
>>>>>>
>>>>>> [/code]
>>>>>>
>>>>>> I've look at the map function of the class "PiEstimator.class" and
it
>>>>>> seems
>>>>>>
>>>>>> ok.
>>>>>>
>>>>>> [code]
>>>>>>
>>>>>> public void map(LongWritable offset,
>>>>>>
>>>>>>         LongWritable size,
>>>>>>
>>>>>>         OutputCollector<BooleanWritable, LongWritable>
out,
>>>>>>
>>>>>>         Reporter reporter) throws IOException {}
>>>>>>
>>>>>> [/code]
>>>>>>
>>>>>>
>>>>>> What's wrong with this examples?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Pedro
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Pedro
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Pedro
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Pedro
>>>
>>
>>
>>
>> --
>> Pedro
>



-- 
Pedro

Mime
View raw message