Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: message received from 54.76.25.247 which
 is an MX secondary for user@hadoop.apache.org)
Message-ID: <553E2D2C.7050702@nissatech.com>
Date: Mon, 27 Apr 2015 14:35:56 +0200
From: Marko Dinic <marko.dinic@nissatech.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.4.0
MIME-Version: 1.0
To: user@hadoop.apache.org
Subject: How to call Hadoop job from a web service in a non-blocking fashion?
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hello,

I have a sequence of jobs that depend on each other, output of one job 
is input for the next one. Also, there is a loop in one part of the 
sequence, containing two jobs executing in a row.

Until now I was able to run this job by simply creating Job objects and 
using waitForCompletition(true). In that way, I was forwarding output of 
one job as input to next one.

The problem is, waitForCompletition(true) will block the web service I'm 
trying to use, so I need a way to run this sequence of dependent jobs, 
but not to get stuck waiting for result of the whole sequence. So, I 
want the next model - user uploads some files, starts the job and gets 
the response that the job has been started. After the sequence has 
finished user should be notified in some way.

I wouldn't like to use Oozie, since this the jobs are more low-level 
(it's actually an algorithm similar to those implemented in Mahout), and 
I don't know if I may use JobControl, since there is a loop, and how to 
do it.

Any help would be highly appreciated.

Regards,
Marko