hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed Radwan (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Date Thu, 29 Mar 2012 16:34:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241368#comment-13241368

Ahmed Radwan commented on MAPREDUCE-3451:

Thanks Patrick.

The patch successfully applies and compiles on trunk. The included new tests also successfully

I'll postpone specific code comments, since I have some basic questions: 

The old fair scheduler used slots as the basic allocation units, and calculating shares and
other features (like preemption) relied on this model. In MR2 we have containers with variable
resource attributes. Currently the only supported attribute is memory, but it is planned to
add more attributes (network bandwidth, cpu, etc.). So it is not clear for me how this mapping
is handled. Are you envisioning combing such attributes for calculating shares, how is this
done? and how preemption will be done in such case, for example in the old MR1, the most recent
slot was killed. But now slots are not equal and this complicate the choices. I am just using
preemption as an example. Can you please elaborate on that?

I also see that the patch doesn't include the web ui components. I think postponing them to
follow-up work is fine. But if someone wants to test the patch on a real cluster, can you
please highlight the steps needed to setup a cluster using the new fair scheduler and if there
are changes in setup compared to the old MR1 version.
> Port Fair Scheduler to MR2
> --------------------------
>                 Key: MAPREDUCE-3451
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2, scheduler
>            Reporter: Patrick Wendell
>            Assignee: Patrick Wendell
>         Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to
MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include
design considerations and progress.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message