hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4053) Schedulers need to know when a job has completed
Date Thu, 25 Sep 2008 10:37:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634434#action_12634434

Hemanth Yamijala commented on HADOOP-4053:

The default implementation for {{jobCompleted}} is calling {{jobRemoved}}. This would result
in {{jobRemoved}} being called twice - once when the job completes, and once when it retires.
Hence all implementations of {{jobRemoved}} need to work correctly even if called twice. For
e.g. in the {{CapacityTaskScheduler}} we clear some data structures. I see that if called
twice, this may result in problems. For e.g. {{qsi.numJobsByUser.get(job.getProfile().getUser());}}
may return null when called a second time, which is not being checked. I think it is safer
to have {{jobCompleted}} similar to other APIs - that is, as an abstract method and implement
it in schedulers. This way, we will be forced to check the implementation is correct.

> Schedulers need to know when a job has completed
> ------------------------------------------------
>                 Key: HADOOP-4053
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4053
>             Project: Hadoop Core
>          Issue Type: Improvement
>    Affects Versions: 0.19.0
>            Reporter: Vivek Ratan
>            Assignee: Amar Kamat
>             Fix For: 0.19.0
>         Attachments: HADOOP-4053-v1.patch
> The JobInProgressListener interface is used by the framework to notify Schedulers of
when jobs are added, removed, or updated. Right now, there is no way for the Scheduler to
know that a job has completed. jobRemoved() is called when a job is retired, which can happen
many hours after a job is actually completed. jobUpdated() is called when a job's priority
is changed. We need to notify a listener when a job has completed (either successfully, or
has failed or been killed). 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message