Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <1742364081.259251263541794551.JavaMail.jira@brutus.apache.org>
Date: Fri, 15 Jan 2010 07:49:54 +0000 (UTC)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Commented: (MAPREDUCE-1379) Limit both numMapTasks and
 numReduceTasks
In-Reply-To: <37277137.257841263537055213.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800605#action_12800605 ] 

Matei Zaharia commented on MAPREDUCE-1379:
------------------------------------------

Is the goal to be able to set "total.tasks.maximum" to less than map.tasks.maximum + reduce.tasks.maximum? If so, you should watch out because you might possibly starve tasks of one type. For example, if you set max map slots to 5, max reduce slots to 5, and max total slots to 5, most of the current job schedulers will probably cause reduce tasks to be starved, because they always look for a map to launch before looking for a reduce. In fact, given that schedulers ultimately decide which tasks to launch anyway, I wouldn't add a "max total slots" concept in the TaskTracker to begin with; I would add this functionality to one of the schedulers.

> Limit both numMapTasks and numReduceTasks
> -----------------------------------------
>
>                 Key: MAPREDUCE-1379
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1379
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: tasktracker
>    Affects Versions: 0.22.0
>            Reporter: Bochun Bai
>            Priority: Trivial
>         Attachments: limit-both-numMapTasks-and-numReduceTasks.patch
>
>
> In some environment, the number of concurrent running process is very sensitive.
>   mapreduce.tasktracker.map.tasks.maximum
> and
>   mapreduce.tasktracker.reduce.tasks.maximum
> limit tasks running on each tasktracker separately.
> This patch limits them together, using mapreduce.tasktracker.total.tasks.maximum

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.