hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3412) Refactor the scheduler out of the JobTracker
Date Tue, 01 Jul 2008 14:02:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609599#action_12609599

Tom White commented on HADOOP-3412:

bq. Separating the JobQueue from the rest of the scheduler right away makes it difficult to
implement certain types of schedulers behind the interface.

I think this is a good point. So I would support the suggestion of having methods to add,
remove and iterate over jobs on the TaskScheduler. And keep JobQueue, but don't make it mandatory
for TaskSchedulers to use it.

Another thing I would like to see is the code for managing the tasktracker state being pushed
up into the base class, since it is shared. (In fact, I wondered whether it belongs in the
TaskScheduler at all, since it would be possible to make TaskScheduler a listener with only
the updateTaskTrackerStatus method. But this would duplicate state with the JobTracker, so
on balance it's fine to have it in the TaskScheduler.)

Taking these points together, we would then have

public abstract class TaskScheduler extends Configured {

  // abstract methods:
  public abstract void addJob(JobInProgress jobInProgress);
  public abstract void removeJob(JobInProgress jobInProgress);
  public abstract Iterator<JobInProgress> getJobsIterator();
  public abstract Task assignTask(String taskTrackerName) throws IOException;

  // base implementations (currently in DefaultTaskScheduler)
  public Collection<TaskTrackerStatus> getAllTaskTrackers() { }
  public int getNumberOfTaskTrackers() { }
  public TaskTrackerStatus getTaskTracker(String taskTrackerName) { }
  public boolean updateTaskTrackerStatus(String taskTrackerName,
      TaskTrackerStatus newStatus) { }
  public Statistics getStatistics() { }

  Collection<JobInProgress> retireOldJobs (long retireBefore) { }


public class DefaultTaskScheduler extends TaskScheduler {

  private JobQueue jobQueue;

  public void addJob(JobInProgress jobInProgress) {

  public Iterator<JobInProgress> getJobsIterator() {
    return jobQueue.iterator();

  public void removeJob(JobInProgress jobInProgress) {

  public Task assignTask(String taskTrackerName) throws IOException {
    // as before


How does this look?

(Note that TaskScheduler can extend Configured rather than implementing Configurable.)

Also, retireOldJobs seems a bit out of place here and should really go back in JobTracker.
This should be easy since it just calls getJobsIterator.

bq. > We don't need iterator() and getSortedJobs() - iterator() is sufficient.

bq. getSortedJobs() allows to bias the choice of the job by the characteristics of the TaskTracker,
something that appeared to be useful when I played with the API. This new proposition however
provides a default implementation for it.

But the iterator returned by iterator() can be any iterator - so we can make it the same one
returned by getSortedJobs(). In other words, we only need one way of iterating over the JobQueue.

> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>                 Key: HADOOP-3412
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3412
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Brice Arnould
>            Assignee: Brice Arnould
>            Priority: Minor
>             Fix For: 0.19.0
>         Attachments: JobScheduler.patch, JobScheduler_v2.patch, JobScheduler_v3.patch,
JobScheduler_v3b.patch, JobScheduler_v4.patch, JobScheduler_v5.patch, JobScheduler_v6.1.patch,
JobScheduler_v6.2.patch, JobScheduler_v6.3.patch, JobScheduler_v6.4.patch, JobScheduler_v6.patch,
JobScheduler_v7.1.patch, JobScheduler_v7.patch, RackAwareJobScheduler.java, SimpleResourceAwareJobScheduler.java
> First I would like warn you that my proposition is assumed to be very naive. I just hope
that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable. But, it is
hard to implement and compare schedulers, because the scheduling logic is mixed within the
rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler. It re-implements
the current scheduling algorithm in a separate class called JobScheduler. This new class is
instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour than the current
JobTracker. More precisely, it doesn't re-implement things like code that seems to be never
called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current implementation, so you
can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you decide about this
patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step of the scheduling.
I hope that it will help future improvement by lowering the level required to contribute to
the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (making it more
> h4. The future
> If you feel that this is a step the right direction, I will try to propose a JobSchedulerInterface
that many JobSchedulers could implement and to propose alternatives to the current « FifoJobScheduler
».  If some of you have ideas about that please tell ^^ I will also open issues for things
marked as FIXME in the patch.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message