hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Date Thu, 29 Mar 2012 17:26:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241407#comment-13241407
] 

Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------

Hey Ahmed,

MR2 does support multiple resource types, but as you point out, currently only memory is implemented.
Right now this scheduler basically treats memory similarly to the way slots were handled in
the old fair scheduler, with the only key difference being that memory can be allocated in
variable size chunks.

There is no widely accepted definition of what "fairness" means with multiple resource types,
but there have been proposals from the academic community (see http://static.usenix.org/events/nsdi11/tech/full_papers/Ghodsi.pdf)
which could ultimately be factored into the scheduler if and when multiple resource types
are introduced. Having several resource dimensions complicates the scheduling decision substantially
and will require major changes to existing schedulers (such as the capacity scheduler) as
well. In all of these cases, existing approaches depend on a single fungible resource about
which you can express minimum capacities (or shares), target capacities (or shares), etc.
These semantics don't make sense with multiple resource types.

Right now the preemption logic preempts lowest priority containers first. This is the best
general-purpose metric I could think of for choosing what to preempt.

I will include a write-up about deployment in the scheduler's current form as well in an upcoming
patch.
                
> Port Fair Scheduler to MR2
> --------------------------
>
>                 Key: MAPREDUCE-3451
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2, scheduler
>            Reporter: Patrick Wendell
>            Assignee: Patrick Wendell
>         Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to
MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include
design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message