hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Dinic <marko.di...@nissatech.com>
Subject How to call Hadoop job from a web service in a non-blocking fashion?
Date Mon, 27 Apr 2015 12:35:56 GMT

I have a sequence of jobs that depend on each other, output of one job 
is input for the next one. Also, there is a loop in one part of the 
sequence, containing two jobs executing in a row.

Until now I was able to run this job by simply creating Job objects and 
using waitForCompletition(true). In that way, I was forwarding output of 
one job as input to next one.

The problem is, waitForCompletition(true) will block the web service I'm 
trying to use, so I need a way to run this sequence of dependent jobs, 
but not to get stuck waiting for result of the whole sequence. So, I 
want the next model - user uploads some files, starts the job and gets 
the response that the job has been started. After the sequence has 
finished user should be notified in some way.

I wouldn't like to use Oozie, since this the jobs are more low-level 
(it's actually an algorithm similar to those implemented in Mahout), and 
I don't know if I may use JobControl, since there is a loop, and how to 
do it.

Any help would be highly appreciated.


View raw message