hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4295) job-level configurable mapred.map.tasks.maximum and mapred.reduce.tasks.maximum
Date Tue, 30 Sep 2008 04:13:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635643#action_12635643
] 

Vinod K V commented on HADOOP-4295:
-----------------------------------

Then, may be, similar to the configuration knob mapred.tasks.maxmemory w.r.t memory, we can
have mapred.job.{map|reduce}.tasks to specify number of tasks a job occupies; while mapred.tasktracker.tasks.maxmemory
maps to mapred.tasktracker.{map|reduce}.tasks.maximium. After that, similar to how HADOOP-4035
wishes to proceed, a scheduler can compare the job's requirements of number of tasks with
tasktracker's limits and scheduler accordingly.

Notes:
 - May we should use the term "cores" in mapred.tasktracker.{map|reduce}.tasks.maximium. We
clearly need to redefine and distinguish tasks, slots and cores, once and for ever.
 - Should we also rename mapred.tasks.maxmemory to mapred.job.tasks.maxmemory?

> job-level configurable mapred.map.tasks.maximum and mapred.reduce.tasks.maximum 
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-4295
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4295
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Christian Kunz
>
> Right now mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum
are set on the tasktracker level.
> In absense of a smart tasktracker monitoring resources and deciding in an adaptive manner
how many tasks can be run simultaneously, it would be nice to move these two configuration
options to the job level. This would make it easier to optimize the performance of a batch
of jobs.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message