hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1198) Alternatively schedule different types of tasks in fair share scheduler
Date Mon, 16 Nov 2009 21:06:39 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778540#action_12778540

Matei Zaharia commented on MAPREDUCE-1198:

Hi Scott,

I don't think what you have is quite the right fix, because when a node has space to launch
more than one task, the scheduler will still assign multiple tasks of the same type rather
than alternating between them. In the pre-0.21 fair scheduler, where only one task of each
type was assigned per heartbeat, this solution would've worked okay. However, in the current
trunk, the scheduler can launch multiple tasks per heartbeat, and with your code, it will
launch many maps or many reduces (depending on which one it had fewer of), but not both.

The right thing would be to add a while loop outside the current for loop. On each iteration
of the while loop, you'd order the task types (as {map, reduce} or {reduce, map}) and then
look for a single task to launch. You then stop the while loop when neither maps nor reduces
have been launched in a particular iteration. That's what I tried to describe in my post on
MAPREDUCE-961, sorry if it wasn't clear.

> Alternatively schedule different types of tasks in fair share scheduler
> -----------------------------------------------------------------------
>                 Key: MAPREDUCE-1198
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1198
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>         Attachments: MAPREDUCE-1198-v1.patch
> Matei has mentioned in MAPREDUCE-961 that the current scheduler will first try to launch
map tasks until canLaunthTask() returns false then look for reduce tasks. This might starve
reduce task. He also mention that alternatively schedule different types of tasks can solve
this problem.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message