hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Date Tue, 29 Nov 2011 23:05:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159619#comment-13159619

Patrick Wendell commented on MAPREDUCE-3451:

I have a few questions about the way that schedulers are implemented in MR2.

It's not clear to me whether the Scheduler class used by the Resource Manager is responsible
for only inter-application scheduling or both inter- ant intra-application scheduling. It
looks to me like only the former is supported, because the resource requests (via allocate())
do not give any meta-data regarding the user initiating the request. So if we wanted to enforce
Fair Sharing for jobs requested from a single AM, this doesn't seem possible. When I look
at the Capacity Scheduler, for instance, it seems like allocation requests are assigned to
queues based only on the application from which they are made, not the actual user launching
the job.

Is the idea that, in a shared cluster, each administrative unit has its own MR Application
Master running? 

I have some follow-up but first want to see if this is actually a correct interpretation of
the design spec.

> Port Fair Scheduler to MR2
> --------------------------
>                 Key: MAPREDUCE-3451
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Patrick Wendell
>            Assignee: Patrick Wendell
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to
MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include
design considerations and progress.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message