hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4295) job-level configurable mapred.map.tasks.maximum and mapred.reduce.tasks.maximum
Date Mon, 29 Sep 2008 03:58:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635280#action_12635280

Hemanth Yamijala commented on HADOOP-4295:

bq. Should the title be changed to something like
bq. Modify the capacity scheduler (HADOOP-3445) to take job limitations concerning number
of simultaneous tasks per node into account when scheduling tasks?

I am not sure if Doug was suggesting we use HADOOP-4035 to implement the functionality proposed
in this JIRA. I understood it to mean that the approach should be the same. Either way, I
think it would be nice to have it handled separately, since HADOOP-4035 is specifically addressing
only memory based parameters in job control.

That said, I also think we'll need to consider unifying mechanisms of resource management
at some time (maybe in the near future, *smile*). We already seem to have *slightly* different
ways of dealing with cores, memory, and disk (a.k.a HADOOP-657) - specifying, measuring, reporting
and scheduling.

> job-level configurable mapred.map.tasks.maximum and mapred.reduce.tasks.maximum 
> --------------------------------------------------------------------------------
>                 Key: HADOOP-4295
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4295
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Christian Kunz
> Right now mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum
are set on the tasktracker level.
> In absense of a smart tasktracker monitoring resources and deciding in an adaptive manner
how many tasks can be run simultaneously, it would be nice to move these two configuration
options to the job level. This would make it easier to optimize the performance of a batch
of jobs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message