hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1129) Assign multiple Map and Reduce tasks in Fairscheduler
Date Thu, 22 Oct 2009 00:23:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768530#action_12768530

Zheng Shao commented on MAPREDUCE-1129:

One simple idea is to change the interface of LoadManager:


boolean LoadManager.canAssignMap(...)
boolean LoadManager.canAssignReduce(...)


int LoadManager.canAssignMapCount(...)
int LoadManager.canAssignReduceCount(...)

The LoadManager returns the number of map(or reduce) tasks that can be assigned to this TaskTracker,
so we can assign multiple tasks in a single heartbeat.
The default implementation of the new functions can directly call the old functions to return
0 or 1, to provide backward compatibility with user-defined LoadManagers.

> Assign multiple Map and Reduce tasks in Fairscheduler
> -----------------------------------------------------
>                 Key: MAPREDUCE-1129
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1129
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.20.1
>            Reporter: Scott Chen
> In Hadoop-0.20, the period of heartbeat becomes much longer. 
> Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
> This makes the cluster become very inefficient. Often time only half of the slots are
> One idea is that we make Fairscheduler detect this situation (cluster under used) and
scheduler more tasks in a heartbeat.
> Thoughts?

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message