hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kai Voigt...@123.org>
Subject Re: Counters in MapReduce
Date Mon, 09 Jun 2014 09:46:35 GMT
Like you said, just wrap your 3 jobs into a while loop and check the built-in counters, like
the number of reduce output records to check if the job output was empty.

Unfortunately, oozie cannot do iterations or loops of jobs, as it only supports DAGs.

Kai

Am 09.06.2014 um 10:31 schrieb unmesha sreeveni <unmeshabiju@gmail.com>:

> I am trying to do iteration with map reduce. I have 3 sequence job running
> 
> //job1 configuration
> FileInputFormat.addInputPath(job1,new Path(args[0]));
> FileOutputFormat.setOutputPath(job1,out1);
> job1.waitForCompletion(true);
> 
> job2 configuration
> FileInputFormat.addInputPath(job2,out1);
> FileOutputFormat.setOutputPath(job2,out2);
> job2.waitForCompletion(true);
> 
> job3 configuration
> FileInputFormat.addInputPath(job3,out2);
> FileOutputFormat.setOutputPath(job3,new Path(args[1]);
> boolean success = job3.waitForCompletion(true);
> return(success ? 0 : 1);
> 
> After job3 I should continue an iteration - job 3 's output should be the input for job1.
And the iteration should continue until the input file is empty. How to accomplish this.
> 
> Will counters do the work. 
> ‚Äč
> 
> -- 
> Thanks & Regards
> 
> Unmesha Sreeveni U.B
> Hadoop, Bigdata Developer
> Center for Cyber Security | Amrita Vishwa Vidyapeetham
> http://www.unmeshasreeveni.blogspot.in/
> 
> 

Kai Voigt			Am Germaniahafen 1			k@123.org
					24143 Kiel					+49 160 96683050
					Germany						@KaiVoigt


Mime
View raw message