hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denim Live <denim.l...@yahoo.com>
Subject Re: naming the output fle of reduce to the partition number
Date Thu, 08 Jul 2010 18:51:31 GMT


yes, I can get the partition number using 
jobconf.getInt("mapred.task.partition") but how can I custom name my output file 
of each reducer with just this partition number?




________________________________
From: Ted Yu <yuzhihong@gmail.com>
To: mapreduce-user@hadoop.apache.org
Sent: Thu, July 8, 2010 6:22:54 PM
Subject: Re: naming the output fle of reduce to the partition number

Please take a look at getUniqueName() method of 
src/mapred/org/apache/hadoop/mapred/FileOutputFormat.java
It retrieves "mapred.task.partition"


On Thu, Jul 8, 2010 at 2:13 AM, Denim Live <denim.live@yahoo.com> wrote:

Hi Everyone,
>I am having some problem with naming the output file of each reduce task with 
>the partition number. First of all, how can  I get the partition number within 
>each reduce? Second, How am I going to name the output file with that partition 
>number? 
>
>I have looked to the MultipleTextOutputFormat. It can generate a new file with 
>the name of my choice for each key. But I want to name the output file for each 
>partition with the name of my choice i.e. with the partition number. 
>
>Please help me in this regard.
> 
>Thanks
>



      
Mime
View raw message