hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4053) Schedulers need to know when a job has completed
Date Sat, 11 Oct 2008 08:34:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638729#action_12638729
] 

Amar Kamat commented on HADOOP-4053:
------------------------------------

I had an offline discussion with Hemanth and Devaraj and here is what we all agreed upon :

bq. Do we need JobStatusChangeEvent(JobInProgress jip, EventType eventType, JobStatus status)
?
No. Job's state is completely manipulated and controlled by an external entity (JobTracker,
JobInProgressListener etc) and hence we should always have an _old_ and a _new_ state. Job's
state changes via well known apis and hence we should capture the state changes whenever these
apis are invoked. Following is the list of knows events w.r.t to a job
||event-type||api||
|Job added|JIP constructor|
|Job inited|JIP.initTasks()|
|Job priority change|JobTracker.setJobPriority()|
|Running Job killed|JobTracker's heartbeat (cleanup tasks)|
|Queued job killed|JobTracker.killJob() --> JIP.terminateJob()|
|Job completed|JobTracker's heartbeat|
|Job removed|JobTracker.finalizeJob()/JobTracker's job expiry thread|

As of now the following changes are tracked 
- job added
- job inited
- job priority change

We just need to track the job-state change in:
- JobTracker.updateTaskStatuses() (job-completion, running-job-kill [external-kill-leading-to-cleanup-tasks,
too-many-failures])
- JobTracker.killJob() (external-queued-job-kill)

bq. Seems OK to have JonInProgress.initTasks() do the event notification
We cant just change the method signature as its a public api. Hence I introduced {{JobInProgress.initTasksAndReportChange()}}.
Looking at Hemanth's comment#2 [here|https://issues.apache.org/jira/browse/HADOOP-4053?focusedCommentId=12638478#action_12638478],
I feel its ok to make the init-event notification inline and whereever needed. We can factor
it out later, if needed.


> Schedulers need to know when a job has completed
> ------------------------------------------------
>
>                 Key: HADOOP-4053
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4053
>             Project: Hadoop Core
>          Issue Type: Improvement
>    Affects Versions: 0.19.0
>            Reporter: Vivek Ratan
>            Assignee: Amar Kamat
>            Priority: Blocker
>         Attachments: HADOOP-4053-v1.patch, HADOOP-4053-v2.patch, HADOOP-4053-v3.1.patch,
HADOOP-4053-v3.2.patch, HADOOP-4053-v4.1.patch, HADOOP-4053-v4.patch
>
>
> The JobInProgressListener interface is used by the framework to notify Schedulers of
when jobs are added, removed, or updated. Right now, there is no way for the Scheduler to
know that a job has completed. jobRemoved() is called when a job is retired, which can happen
many hours after a job is actually completed. jobUpdated() is called when a job's priority
is changed. We need to notify a listener when a job has completed (either successfully, or
has failed or been killed). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message