hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: Cyclic dependency in JobControl job DAG
Date Mon, 25 Jun 2012 13:53:30 GMT
I personally think it is useful.  I would say contribute it.

(Moved common-dev to bcc, we try not to cross post on these lists)

--Bobby Evans

On 6/25/12 3:37 AM, "madhu phatak" <phatak.dev@gmail.com> wrote:

Hi,
 In current implementation of JobControl, whenever there is a cyclic
dependency between the jobs it throws a Stack overflow exception .
 For example,
   ControlledJob job1 = new ControlledJob(new Configuration());
job1.setJobName("job1");
ControlledJob job2 = new ControlledJob(new Configuration());
job2.setJobName("job2");
job1.addDependingJob(job2);
job2.addDependingJob(job1);
                JobControl jobControl = new JobControl("jobcontrol");
jobControl.addJob(job1);
jobControl.addJob(job2);
jobControl.run();

throws
  java.lang.StackOverflowError
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.checkState(ControlledJob.java:295)

Whenever we write complex application, there is always possibility of
cyclic dependencies.I have written a method which checks for the cyclic
dependency  upfront and informs it to the user. I want to know from you
guys, do you think is it a useful feature? If yes I can contribute it as a
patch.

Regards,
Madhukara Phatak
--
https://github.com/zinnia-phatak-dev/Nectar


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message