hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jkupferman <jkupfer...@umail.ucsb.edu>
Subject Re: Getting map ouput as final output by setting number of reduce to zero
Date Wed, 30 Apr 2008 22:47:07 GMT

You actually dont need to do that, you can just use an IdentityReducer. The
IdentityReducer simply acts as a pass through, so whatever is written out
from your mapper, is passed through it and then written out to file. 


vibhooti wrote:
> 
> Has any one tried setting number of reduce to zero and getting map's
> output
> as the final output?
> I tried doing the same but my map output does not come to specified output
> path for mapred.
> let me know if someone has already done that. I am not able to find out,
> where my map outputs are written.
> http://hadoop.apache.org/core/docs/r0.16.3/mapred_tutorial.html say the
> following
> 
> Reducer NONE
> 
> It is legal to set the number of reduce-tasks to *zero* if no reduction is
> desired.
> 
> In this case the outputs of the map-tasks go directly to the FileSystem,
> into the output path set by
> setOutputPath(Path)<http://hadoop.apache.org/core/docs/r0.16.3/api/org/apache/hadoop/mapred/JobConf.html#setOutputPath%28org.apache.hadoop.fs.Path%29>.
> The framework does not sort the map-outputs before writing them out to the
> FileSystem.
> 
> 
> -- 
> cheers,
> Vibhooti
> 
> 

-- 
View this message in context: http://www.nabble.com/Getting-map-ouput-as-final-output-by-setting-number-of-reduce-to-zero-tp16847897p16992742.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


Mime
View raw message