hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1379) Limit both numMapTasks and numReduceTasks
Date Fri, 15 Jan 2010 07:49:54 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800605#action_12800605
] 

Matei Zaharia commented on MAPREDUCE-1379:
------------------------------------------

Is the goal to be able to set "total.tasks.maximum" to less than map.tasks.maximum + reduce.tasks.maximum?
If so, you should watch out because you might possibly starve tasks of one type. For example,
if you set max map slots to 5, max reduce slots to 5, and max total slots to 5, most of the
current job schedulers will probably cause reduce tasks to be starved, because they always
look for a map to launch before looking for a reduce. In fact, given that schedulers ultimately
decide which tasks to launch anyway, I wouldn't add a "max total slots" concept in the TaskTracker
to begin with; I would add this functionality to one of the schedulers.

> Limit both numMapTasks and numReduceTasks
> -----------------------------------------
>
>                 Key: MAPREDUCE-1379
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1379
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: tasktracker
>    Affects Versions: 0.22.0
>            Reporter: Bochun Bai
>            Priority: Trivial
>         Attachments: limit-both-numMapTasks-and-numReduceTasks.patch
>
>
> In some environment, the number of concurrent running process is very sensitive.
>   mapreduce.tasktracker.map.tasks.maximum
> and
>   mapreduce.tasktracker.reduce.tasks.maximum
> limit tasks running on each tasktracker separately.
> This patch limits them together, using mapreduce.tasktracker.total.tasks.maximum

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message