hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Assigned) (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (MAPREDUCE-3210) Support delay scheduling for node locality in MR2's capacity scheduler
Date Thu, 20 Oct 2011 22:24:11 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Todd Lipcon reassigned MAPREDUCE-3210:
--------------------------------------

    Assignee:     (was: Todd Lipcon)

Turns out the major locality issues I was seeing were related to data locality not being respected
at all. This was fixed by MAPREDUCE-2693 (see also MAPREDUCE-3234)
                
> Support delay scheduling for node locality in MR2's capacity scheduler
> ----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3210
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3210
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>
> The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level
locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially
on clusters where disk throughput is much better than network capacity, this hurts overall
job performance. We should optionally support node-level delay scheduling heuristics similar
to what the fair scheduler implements in MR1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message