hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3210) Support delay scheduling for node locality in MR2's capacity scheduler
Date Tue, 03 Jan 2012 04:54:22 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178627#comment-13178627
] 

Patrick Wendell commented on MAPREDUCE-3210:
--------------------------------------------

I'm going to be addressing this as part of MAPREDUCE-3601 and can probably just add to the
Capacity scheduler as well. 

Delay scheduling is going to be less efficient in MR2 due to the resource request model. Right
now, when a map task needs to run, the MR AM creates three separate resource requests to the
scheduler, one for a node-local container, one for a rack-local container, and another for
an *any* container. However, the scheduler can't associate these in any way.

In the MR1 Fair scheduler, we basically triage a given request and accept worse levels of
locality as time goes on - this won't be possible. In MR2, I don't see a better way than introducing
some type of global delay for "any" requests and rack-local requests (the former exists already).
It seems like this could lead to undesirable behaviour depending on the order of resource
request arrivals.
                
> Support delay scheduling for node locality in MR2's capacity scheduler
> ----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3210
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3210
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>
> The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level
locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially
on clusters where disk throughput is much better than network capacity, this hurts overall
job performance. We should optionally support node-level delay scheduling heuristics similar
to what the fair scheduler implements in MR1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message