hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4053) Schedulers need to know when a job has completed
Date Fri, 10 Oct 2008 04:44:53 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638478#action_12638478

Hemanth Yamijala commented on HADOOP-4053:

We are getting close. But I missed two points from the last patch which I think are important.
Sorry !

- Do we need {{JobStatusChangeEvent(JobInProgress jip, EventType eventType, JobStatus status)}}
? It is confusing to set {{oldStatus}} and {{newStatus}} to the same object. It might be much
better to say that {{oldStatus}} is null and not available, but that's also confusing, for
why would it not be available. I think the main reason is that in {{JobTracker.finalizeJob}},
maybe by the time we can raise the event, the status is already changed. If that's the case,
we should actually try and clone the status before changing. This may be a bit more code,
but it seems clearer to keep the model that {{JobStatusChangeEvent}} always has a correct
{{oldStatus}} and {{newStatus}}.

- I don't think {{initTasksAndReportChange}} should be a public API. It seems like a helpful
wrapper - that's all. Also, I'm not sure if what it's doing is 'reporting' a change. In fact,
it seems to me that it is better to have this wrapper API where needed. Currently, I think
it is in {{JobTracker}} and {{CapacityTaskScheduler}}. This is a little code duplication,
but since the choice of raising the event in this case is flexible, it seems OK to have this

> Schedulers need to know when a job has completed
> ------------------------------------------------
>                 Key: HADOOP-4053
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4053
>             Project: Hadoop Core
>          Issue Type: Improvement
>    Affects Versions: 0.19.0
>            Reporter: Vivek Ratan
>            Assignee: Amar Kamat
>            Priority: Blocker
>         Attachments: HADOOP-4053-v1.patch, HADOOP-4053-v2.patch, HADOOP-4053-v3.1.patch,
HADOOP-4053-v3.2.patch, HADOOP-4053-v4.1.patch, HADOOP-4053-v4.patch
> The JobInProgressListener interface is used by the framework to notify Schedulers of
when jobs are added, removed, or updated. Right now, there is no way for the Scheduler to
know that a job has completed. jobRemoved() is called when a job is retired, which can happen
many hours after a job is actually completed. jobUpdated() is called when a job's priority
is changed. We need to notify a listener when a job has completed (either successfully, or
has failed or been killed). 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message