hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-5199) A proposal to merge common functionality of various Schedulers
Date Mon, 09 Feb 2009 13:20:59 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Vivek Ratan updated HADOOP-5199:

    Attachment: 5199.1.patch

I'm attaching a patch (5199.1.patch) that illustrates some of the proposals I made in the
previous comment. In this patch: 
* {{HadoopTaskScheduler}} implements the Hadoop Scheduler. This class extends {{TaskScheduler}}.
Anyone who wants to write their own scheduler can continue to extend {{TaskScheduler}}. 
* I have copied taken the {{Pool}} and {{PoolManager}} classes from FS, and copied them to
src/mapred/o/a/h/mapred and renamed them to {{Pool2}} and {{Pool2Manager}}. (If this proposal
works, FS's {{Pool}} and {{Pool2Manager}} classes will be deleted and {{Pool2}} and {{Pool2Manager}}
will be renamed to their original names. {{Pool2}} represents a pool - a collection - of jobs,
while {{Pool2Manager}} simply provides some methods to manage pools (and, perhaps, can be
* I needed to support the fact that we may have different ordering of jobs (as described in
the earlier comment). To do this, I have made the {{Pool2}} class abstract. Developers are
expected to extend this class to provide their own ordering of jobs in a pool. I can think
of two sub-classes of {{Pool2}}: one which orders jobs based on priority and job submission
time (which does what CS and DS want), and one which orders jobs based on fairshare criteria.
I have implemented the former as the {{PriorityPool}} class, using the code from {{JobQueueJobScheduler}}.
The benefit of this approach is that anyone can provide their own ordering of jobs and simply
implement the ordering code. Everything else is handled by the {{Pool2}} base class. When
{{Pool2Manager}} creates {{Pool2}} objects, it actually creates {{PriorityPool}} classes (or
any other sub-class of {{Pool2}}) as configured. 
* When {{HadoopTaskScheduler}} starts up, it reads pool information from a pool configuration
file using the {{PoolConf}} class. This class expects configuration to be in the same Hadoop
format as other config files, rather than in XML format as used by FS. Each pool is configured
with a capacity (which I refer to as 'guaranteed capacity' in the code). This should probably
be a percentage value and we should do the same sanity checks as we did in CS. Pools will
also be configured with user limits, which I haven't implemented yet. 
* I have not implemented user limits yet. We can presumably have user limits based on tasks
(each user can have no more than X% of the pool's capacity in terms of running tasks, as CS
does), or based on jobs (each user can have no more than X% or Y jobs running at any given
time) similar to what FS does. 
* I have not implemented memory-based scheduling, but it has been implemented in CS, and can
be copied over with minimal fuss, I suspect. 

I'd love to get feedback and thoughts from folks. This patch still needs work and is meant
to illustrate some of the proposals I had made earlier. 

> A proposal to merge common functionality of various Schedulers
> --------------------------------------------------------------
>                 Key: HADOOP-5199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5199
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Vivek Ratan
>         Attachments: 5199.1.patch
> There are at least 3 Schedulers in Hadoop today: Default, Capacity, and Fairshare. Over
time, we're seeing a lot of functionality common to all three. Many bug fixes, improvements
to existing functionality, and new functionality are applicable to all three schedulers. This
trend seems to be getting stronger, as we notice similar problems, solutions, and ideas. This
is a proposal to detect and consolidate such common functionality, while at the same time,
allowing for differences in behavior and leaving the door open for other schedulers to be
built, as per HADOOP-3412. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message