hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (MAPREDUCE-1859) maxConcurrentMapTask & maxConcurrentReduceTask per job
Date Wed, 30 Jul 2014 20:16:39 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Allen Wittenauer resolved MAPREDUCE-1859.
-----------------------------------------

    Resolution: Won't Fix

Closing as won't fix.  With YARN, you don't need to do these types of operations in MR anymore.

> maxConcurrentMapTask & maxConcurrentReduceTask per job
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-1859
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1859
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: job submission
>    Affects Versions: 0.20.2
>            Reporter: Johannes Zillmann
>
> It would be valuable if one could specify the max number of map/reduce slots which should
be used for a given job. An example would be an map-reduce job importing from a database where
you don't want 50 map tasks querying one db at a time but also you don't want to shrink the
overall map task count.
> Also this is probably already possible through Fair/Capacity-Scheduler or an own Extension
i think it would be a good addition for the default TaskScheduler since this seems to be more
then a rare used feature.
> This would have the benefit in situations where you don't have control/ownership over
the cluster as well. 
> And its more job-centric whereas the existing scheduler extensions seems to be more job-type-centric.
> Implementing this feature should be relatively straightforward. Adding something like
jobConf.setMaxConcurrentMapTask(int) and respecting this configuration in JobQueueTaskScheduler.
> Not sure if this feature would be harmonical with the existing Fair/Capacity-Schedulers.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message