hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Parker <michael.g.par...@gmail.com>
Subject Two-stage MR not terminating gracefully with JobControl?
Date Sun, 19 Aug 2012 16:46:15 GMT
Hi all,

I have a two-stage MR, written using entirely the new API, that I'm
running using Hadoop 1.0.3. In the end it generates a file named
part-r-0000 with the correct output, but the main method doesn't
terminate, despite me using the allFinished method of JobControl as I
thought I was supposed to. Any help would be greatly appreciated. A
snippet from my code follows.


import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;


public static void main(String[] args) throws IOException {

  Job job1 = new Job(new Configuration(conf), "job1");
  ControlledJob controlledJob1 = new ControlledJob(job1,

  Job job2 = new Job(new Configuration(conf), "job2");
  ControlledJob controlledJob2 = new ControlledJob(job2,

  JobControl jobControl = new JobControl("main");
  jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
  Thread t = new Thread(jobControl);
  while (!jobControl.allFinished()) {
    try {
    } catch (InterruptedException e) {
      // Ignore.



- Mike

View raw message