hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3412) Refactor the scheduler out of the JobTracker
Date Mon, 30 Jun 2008 22:58:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609397#action_12609397

Matei Zaharia commented on HADOOP-3412:

Hi Brice,

You're right that the JobQueue abstraction is very nice and that a lot  
of people would want to use it. However, I'm just saying that when  
defining the interface to the scheduler, it should be as simple and  
versatile as possible. Part of the goal of this JIRA was to make it  
possible for organizations to control scheduling without modifying  
Hadoop source code, so the interface it defines for the scheduler  
should hopefully never have to change once it is committed. Separating  
the JobQueue from the rest of the scheduler right away makes it  
difficult to implement certain types of schedulers behind the  
interface. Instead, I'm suggesting that you just add the JobQueue  
functionality to DefaultTaskScheduler (or a superclass, maybe  
something called QueueBasedTaskScheduler). Users will still be able to  
use JobQueue in the same way (just specify it in a config file), but  
anyone wanting to do something more complicated will have a simple  
interface to implement (just TaskScheduler). This is less "hacky" than  
creating a class that is both a JobQueue and TaskScheduler.

The reason I bring this up is because I've been trying to use your  
patch to write a scheduler that emulates what Hadoop on Demand does  
today - the ability to specify quotas for each user (a max number of  
slots that the user can use). The most elegant way to implement this  
would be to have a TaskScheduler that knows about a quota on each  
user, and contains a separate JobQueue for each user, maybe chosen by  
the user (some might want FIFO, some might want shortest job first,  
etc). However, with the current interface, I have to create a single  
JobQueue class and hack the getSortedJobs method to return different  
things based on how many tasks each user has running. In fact changing  
the interface to TaskScheduler to make it more general as suggested  
would allow *more* code reuse because I'd be able to reuse other  
peoples' JobQueues as long as I write a TaskScheduler that works with  
multiple queues.


> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>                 Key: HADOOP-3412
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3412
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Brice Arnould
>            Assignee: Brice Arnould
>            Priority: Minor
>             Fix For: 0.19.0
>         Attachments: JobScheduler.patch, JobScheduler_v2.patch, JobScheduler_v3.patch,
JobScheduler_v3b.patch, JobScheduler_v4.patch, JobScheduler_v5.patch, JobScheduler_v6.1.patch,
JobScheduler_v6.2.patch, JobScheduler_v6.3.patch, JobScheduler_v6.4.patch, JobScheduler_v6.patch,
JobScheduler_v7.1.patch, JobScheduler_v7.patch, RackAwareJobScheduler.java, SimpleResourceAwareJobScheduler.java
> First I would like warn you that my proposition is assumed to be very naive. I just hope
that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable. But, it is
hard to implement and compare schedulers, because the scheduling logic is mixed within the
rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler. It re-implements
the current scheduling algorithm in a separate class called JobScheduler. This new class is
instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour than the current
JobTracker. More precisely, it doesn't re-implement things like code that seems to be never
called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current implementation, so you
can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you decide about this
patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step of the scheduling.
I hope that it will help future improvement by lowering the level required to contribute to
the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (making it more
> h4. The future
> If you feel that this is a step the right direction, I will try to propose a JobSchedulerInterface
that many JobSchedulers could implement and to propose alternatives to the current « FifoJobScheduler
».  If some of you have ideas about that please tell ^^ I will also open issues for things
marked as FIXME in the patch.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message