hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1111) Job completion notification to a job configured URL
Date Tue, 13 Mar 2007 10:28:09 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alejandro Abdelnur updated HADOOP-1111:
---------------------------------------

    Attachment: patch-1111.txt

Patch with indentation of 2 and max line len of 80.

Tom,

The problem with the hadoop retry framework (if I understand correctly how it works) is that
the policy goes to sleep synchronously within from which is invoked.

It could be used for the LocalJobRunner but for the JobTracker it would block it while both
sending the notification and waiting for the retries. 

With the producer-consumer queue being consumed by a side thread no blocking happens in the
JobTracker job handling logic.

What am I missing?

A

> Job completion notification to a job configured URL
> ---------------------------------------------------
>
>                 Key: HADOOP-1111
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1111
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.12.0
>         Environment: all
>            Reporter: Alejandro Abdelnur
>         Attachments: patch-1111.txt, patch-1111.txt
>
>
> Currently clients have to poll the JobTracker to find if a job has completed or not.
> When invoking Hadoop from other systems is desirable to have a notification mechanism
on job completion. 
> The notification approach simplifies the client waiting for completion and removes load
from the JobTracker as polling can be avoided. 
> Proposed solution:
> When the JobTracker processes the completion of a job (success and failure)  if the job
configuration has a jobEnd.notificationUrl property it will make a HTTP GET request to the
specified URL.
> The jobEnd.notificationUrl property may include 2 variables in it '${jobId}' and '${jobStatus}'.
if they are present, they will be replaced with tehe job ID and status of the job and the
URL will be invoked.
> Two additional properties, 'jobEnd.retries' and 'jobEnd.retryInterval', will indicate
retry behavior.
> Not to delay the JobTracker processing while doing notifications, a ConsumerProducer
Queue will be used to queue up job notification upon completion.
> A daemon thread will consume job notifications from the above Queue and will make the
URL invocation. 
> On notification failure, the job notification is  queue up again on the notification
queue.
> The queue will be a java.util.concurrent.DelayQueue. This will make job notifications
(on retries) to be avaiable on the consumer side only when the retry time is up.
> The changes will be done in the JobTracker and in the LocalJobRunner.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message