hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <ar...@yahoo-inc.com>
Subject Re: Q: Sending output of reduce to mapper
Date Sun, 07 Oct 2007 12:24:44 GMT
Hi Ken,

On Sat, Oct 06, 2007 at 08:54:54PM -0700, Ken Pu wrote:
>As a beginner of Hadoop, I wonder how to send output key-value pairs of the
>reducers back to the input of mappers for iterative processing.  

A map-reduce job has only 1 set of maps and 1 set of reduces.

The way to do what you seek would be to chain jobs together i.e. output of job1 becomes input
of job2 and so on. That is fairly easy since the output of the job (i.e. reduces) is on hdfs,

Clearly the onus on waiting for job-completion is on the user-code i.e. you have to ensure
job1 is complete before launching job2 and so on...

The way to do that would be: 
a) http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/JobClient.html#runJob(org.apache.hadoop.mapred.JobConf)
which submits the job and returns only after it completes (success or failure).
b) http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/JobClient.html#submitJob(org.apache.hadoop.mapred.JobConf)
to just submit the job and poll yourself, look at src/java/org/apache/hadoop/mapred/JobClient.java
and in particular the implmentation of *runJob* on how to do taht.
b) If you don't want to poll use the *job.end.notification.url* property where you can setup
a url which will be invoked once the job completes to do async-stuff. (Take a look at src/test/org/apache/hadoop/mapred/NotificationTestCase.java
for an e.g. on how to use that).

>What's hadoop streaming?  Can I pipe the output stream of reducers back to
>the input stream of the mappers to achieve what I want?

Hadoop streaming is a utility which allows the user to create and run map/reduce jobs with
any executables as the mapper and/or the reducer. 
E.g. one can use std. unix utilities as the mapper/reducer
$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar
  -input myInputDirs \
  -output myOutputDir \
  -mapper /bin/cat \
  -reducer /bin/wc

Hope that helps.


>Any pointer would be greatly appreciated.
>View this message in context: http://www.nabble.com/Q%3A-Sending-output-of-reduce-to-mapper-tf4581957.html#a13079722
>Sent from the Hadoop Users mailing list archive at Nabble.com.

View raw message