hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengwei Wang (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (MAPREDUCE-7274) Enable to limit running map and reduce tasks when job is running
Date Tue, 26 May 2020 07:09:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084662#comment-17084662
] 

Chengwei Wang edited comment on MAPREDUCE-7274 at 5/26/20, 7:08 AM:
--------------------------------------------------------------------

upload patch for branch 3.2.0. 

It enable us to set limit for running tasks when job is running by mapred client command:
{code:bash}
mapred job -set-running-task-limit JOB_ID TASK_TYPE LIMIT

e.g.
   mapred job -set-running-task-limit job_1583809537551_21297 MAP 100
   mapred job -set-running-task-limit job_1583809537551_21297 REDUCE 100
{code}
it send a rpc resquest to AM, and AM would update the max running limit of the specified
type task as specified limit count.

 
  


was (Author: smarthan):
upload patch for branch 3.2.0.

 

It enable us to set limit for running tasks when job is running by mapred client command:
{code:bash}
mapred job -set-running-task-limit JOB_ID TASK_TYPE LIMIT

e.g.
   mapred job -set-running-task-limit job_1583809537551_21297 MAP 100
   mapred job -set-running-task-limit job_1583809537551_21297 REDUCE 100
{code}
it send a rpc resquest to AM, and AM would update the max running limit of the specified
type task as specified limit count.

 
  

> Enable to limit running map and reduce tasks when job is running
> ----------------------------------------------------------------
>
>                 Key: MAPREDUCE-7274
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7274
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, mrv2
>            Reporter: Chengwei Wang
>            Priority: Major
>         Attachments: MAPREDUCE-7274.patch
>
>
> MRv2 enabled users to control the number of map or reduce tasks running simultaneously
by configuration  *_mapreduce.job.running.map.limit_* or _*mapreduce.job.running.reduce.limit*._
But users can only set limit number before submitting the job to rm. So, it's meaningful 
to enable users to set the limit of running map or reduce tasks when job is running, which
can help users to restrict  resource usage of job and give resources to high-priority job.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message