hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sarthak Dudhara <sarthak...@gmail.com>
Subject Re: 0.20.2 : Running Chained jobs using JobControl
Date Sun, 02 Jan 2011 15:49:45 GMT
Hi Harsh,

Yes exactly, simple chaining could work in this case. However, we envision
complex workflows in the future and so we would checkout other tooling
available for that.

Sarthak Dudhara



On Sun, Jan 2, 2011 at 3:06 AM, Harsh J <qwertymaniac@gmail.com> wrote:

> Although for complex workflows one should checkout Oozie or Azkaban.
>
> On Sun, Jan 2, 2011 at 1:55 PM, Hari Sreekumar <hsreekumar@clickable.com>
> wrote:
> > Can't we run chained jobs like this?
> > bool j1 = job1.waitForCompletion(..) ;
> > if(b1) job2.waitForCompletion(..) ;
> >
> > and setting up the jobs such that job1's output dir is job2's input dir?
> > Thanks,
> > Hari
>
> Yes, this could work for simple success/failure based chaining
> (although it makes the driver code look a tad messy?).
>
> This is what JobControl is aiming to provide from within Hadoop
> libraries itself. Plus the ability to have more control on the
> dependent, waiting jobs.
>
> --
> Harsh J
> www.harshj.com
>

Mime
View raw message