hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ioan Eugen Stan <stan.ieu...@gmail.com>
Subject Re: Fw: reducers outputs
Date Mon, 30 Jan 2012 08:01:15 GMT
Pe 30.01.2012 07:47, aliyeh saeedi a scris:
>
>
>
>
>
>
> I want to save them with my own names, How NameNode will keep their names?
>
>
>
> ________________________________
>   From: Joey Echeverria<joey@cloudera.com>
> To: mapreduce-user@hadoop.apache.org; aliyeh saeedi<a1_saeedi@yahoo.com>
> Sent: Sunday, 29 January 2012, 17:10
> Subject: Re: reducers outputs
>
> Reduce output is normally stored in HDFS, just like your other files.
> Are you seeing
>   different behavior?
>
> -Joey
>
> On Sun, Jan 29, 2012 at 1:05 AM, aliyeh saeedi<a1_saeedi@yahoo.com>  wrote:
>> Hi
>> I want to save reducers outputs like other files in Hadoop. Does NameNode
>> keep any information about them? How can I do this?
>> Or can I add a new component to Hadoop like NameNode and make JobTracker to
>> consult with it too (I mean I want to make JobTracker to consult with
>> NameNode AND myNewComponent both)?
>
>
>

You aren't making a lot of sens, at least to me :). But if tou wish to 
save reducer output somehow different you will have to implement your 
own class that implements 
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/OutputFormat.html.

It's easier to extend 
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/FileOutputFormat.html

and override the parts that you need.

The framework will call methods from the above mentioned class to 
persist the data from reducers. You instruct the framework to use your 
class when you call 
job.setOutputFormatClass(SequenceFileOutputFormat.class); (this makes 
the output a SequenceFile).

Example to save under a different name:

public static class RenamedSequenceFile extends SequenceFileOutputFormat {

         @Override
         public Path getDefaultWorkFile(TaskAttemptContext context, 
String extension) throws IOException {
             FileOutputCommitter committer = (FileOutputCommitter) 
getOutputCommitter(context);
             return new Path(committer.getWorkPath(), "myBetterName");
         }
     }

This will output your reducer data into "myBetterName" file as key 
values pairs (behaviour inherited from SequanceFileOutputFormat).

I hope this helps,

-- 
Ioan Eugen Stan
http://ieugen.blogspot.com

Mime
View raw message