hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4053) Schedulers need to know when a job has completed
Date Fri, 26 Sep 2008 11:29:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634820#action_12634820

Hemanth Yamijala commented on HADOOP-4053:

Owen, the jobUpdated API currently passes the JobInProgress argument. If this is called at
every state change, the listeners would need to keep track of what actually changed. I think
that would be a bit expensive, because it would mean the schedulers need to keep a copy of
the old state doubling the memory requirements. 

A better API would have been something like {{jobStateChanged(ChangeEvent e)}} where the ChangeEvent
captures the state transition in some way. This would keep it generic and allow for additional
state transitions to be added in the future. If done this way, it would solve the problem.

Looking at the way it currently is implemented, it seems like the intent is to capture each
state change via a separate call, and somehow the job completion event was missed (or thought
unnecessary). In that case, possibly adding jobCompleted is the right approach. This does
have the problem that additional state changes that may come in future (for e.g. paused ?)
may need changes to the interface again.

So, either way, I think we would need to change the current interface a bit. And it seems
like adding jobCompleted is less intrusive at this stage. Comments ?

> Schedulers need to know when a job has completed
> ------------------------------------------------
>                 Key: HADOOP-4053
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4053
>             Project: Hadoop Core
>          Issue Type: Improvement
>    Affects Versions: 0.19.0
>            Reporter: Vivek Ratan
>            Assignee: Amar Kamat
>             Fix For: 0.19.0
>         Attachments: HADOOP-4053-v1.patch, HADOOP-4053-v2.patch
> The JobInProgressListener interface is used by the framework to notify Schedulers of
when jobs are added, removed, or updated. Right now, there is no way for the Scheduler to
know that a job has completed. jobRemoved() is called when a job is retired, which can happen
many hours after a job is actually completed. jobUpdated() is called when a job's priority
is changed. We need to notify a listener when a job has completed (either successfully, or
has failed or been killed). 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message