hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy <snickerdoodl...@gmail.com>
Subject setting a different input/output class for combiner function than map and reduce functions
Date Wed, 24 Sep 2008 00:40:40 GMT
Hi,

I'm trying to refine my map reduce algorithm to run faster, but I ran into a
little bit of trouble.

In my main, I have the following parameters set for my conf:

conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(LongWritable.class);


originally, this was quite ok, since my input to my map phase was key =
LongWritable value = Text,
and the output of my map phase was key = Text, value = LongWritable.

Similarily, the input to my reduce phase was key = Text, value =
iterator<LongWritable>
and the output was key = Text, value = LongWritable.

I just wrote a combiner class to try and speed things up. However, now I
want to do something like the following:
==map phase==
input: key = LongWritable value = Text,
output: key = Text, value = Longwritable

==combiner==
input: key = Text, value = iterator<LongWritable>
output: key = Text, value = Text

==reduce phase==
input: key = Text, value = iterator<Text>
output: key = Text, value = LongWritable


According to the documentation I saw, I shouldn't have to change what I have
set in main(), since the key type of the output of the job is Text, and the
value type is still LongWritable.

However, when I compile and try to run the code, I get the following:
08/09/23 18:35:45 INFO mapred.JobClient: Task Id :
task_200809222115_0004_m_000000_0, Status : FAILED
java.io.IOException: wrong value class: org.apache.hadoop.io.Text is not
class org.apache.hadoop.io.LongWritable

This tells me that Hadop doesn't like the fact that my output at the end of
the combiner is of type Text, which also tells me that I mustn't have set
the proper confs or something along those lines. However, I don't see
anything in the JobConf class that would let me set the output of the
combiner. I see a setMapOutputValueClass, but no setCombinerOutputValueClass
function.

Could someone point me in the right direction please?

Thanks in advance,

-SM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message