hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5199) A proposal to merge common functionality of various Schedulers
Date Tue, 10 Feb 2009 07:16:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672186#action_12672186
] 

Vivek Ratan commented on HADOOP-5199:
-------------------------------------

It may not be clear from what I've written so far, but I'm *not* proposing yet another scheduler.
I'm suggesting that there is a large amount of common code/functionality in the three existing
schedulers today, which we can combine into one common base scheduler (which I refer to as
the 'Hadoop Scheduler'). Features such as detecting how many M/R tasks to allocate in one
heartbeat, pools and capacities, getting one or more node-local map tasks, checking for user
limits and memory requirements, etc. are all common functionality that this base scheduler
can implement in one place. At the same time, I'm suggesting that we have two different ways
of ordering jobs (one matching what Fairshare does, the other similar to the default and Capacity
schedulers). This design (the  base scheduler and the two ways of ordering jobs) is intended
to replace _contrib/capacity-scheduler_ and _contrib/fairscheduler_ while making sure we support
as much of existing functionality as possible. Putting common code in one place will help
efforts such as global scheduling and others that impact all schedulers, as there is only
one set of code to fix. 

We've had some discussions in the past, across various Jiras, to consolidate common code/functionality
and this is an attempt to do so. 

> A proposal to merge common functionality of various Schedulers
> --------------------------------------------------------------
>
>                 Key: HADOOP-5199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5199
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Vivek Ratan
>         Attachments: 5199.1.patch
>
>
> There are at least 3 Schedulers in Hadoop today: Default, Capacity, and Fairshare. Over
time, we're seeing a lot of functionality common to all three. Many bug fixes, improvements
to existing functionality, and new functionality are applicable to all three schedulers. This
trend seems to be getting stronger, as we notice similar problems, solutions, and ideas. This
is a proposal to detect and consolidate such common functionality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message