hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arko Provo Mukherjee <arkoprovomukher...@gmail.com>
Subject Re: Reducers without output files
Date Thu, 15 Sep 2011 05:16:27 GMT
Hello,

Many thanks for your reply!

So to clarify, I should do the following:
public static class Reduce extends MapReduceBase implements
Reducer<IntWritable, Text, NullWritable, Text> {

reduce ()  {  // Pseudo reduce funtion - ignoring the proper syntax

// The processing goes here.
output.collect ( new NullWritable(), new Text(output_string) );


}

}

Finally in the main method of the Driver Class:

// For the Map Class
jobconf.setMapOutputKeyClass(IntWritable.class);
jobconf.setMapOutputValueClass(Text.class);

// For the Reduce Class
jobconf.setOutputKeyClass(NullWritable.class);
jobconf.setOutputValueClass(Text.class);

Please do correct me if my understanding is wrong.

Thanks again for your help!

Warm Regards
Arko

On Wed, Sep 14, 2011 at 11:12 PM,  <bejoy.hadoop@gmail.com> wrote:
> Akro
>     To add on, if you want to ignore  the key part then substitute key with NullWritable.
And do the desired modification in driver class in place of output key type.
>
> Hope it helps
> Regards
> Bejoy K S
>
> -----Original Message-----
> From: bejoy.hadoop@gmail.com
> Date: Thu, 15 Sep 2011 04:09:12
> To: <mapreduce-user@hadoop.apache.org>
> Reply-To: bejoy.hadoop@gmail.com
> Subject: Re: Reducers without output files
>
> Hi Akro
>       You can achieve the same within the existing mapreduce frame work itself. Give
a NullWritable in place of reducer output value in reduce function. In your driver class as
well mention the output value type as NullWritable.
>
> ------Original Message------
> From: Arko Provo Mukherjee
> To: mapreduce-user@hadoop.apache.org
> ReplyTo: mapreduce-user@hadoop.apache.org
> Subject: Reducers without output files
> Sent: Sep 15, 2011 08:56
>
> Hello Everyone,
>
> I have a small issue with my Reducer that I am trying to figure out
> and wanted some advice.
>
> In the reducer, when writing to the output file as declared in
> FileOutputFormat.setOutputPath() I want to write only the key and not
> the value when I am calling output.collect().
>
> Is there a way I can ignore the key part?
>
> Else,
>
> Can I write a Reducer function that doesn't do a output.collect()??
>
> Say I omit the FileOutputFormat.setOutputPath() in the Driver Class.
>
> I can then manually write the output to HDFS in the format I like.
>
> Is this legal way to do stuff?
>
> Many thanks in advance!
> Warm Regards
> Arko
>
>
> Regards
> Bejoy K S

Mime
View raw message