hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (Jira)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-7274) Enable to limit running map and reduce tasks when job is running
Date Tue, 26 May 2020 10:28:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116609#comment-17116609
] 

Hadoop QA commented on MAPREDUCE-7274:
--------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 41s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  0s{color} |
{color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue}  0m  0s{color} | {color:blue}
prototool was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 4 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  8s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 15s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 55s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 54s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 57s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 52s{color}
| {color:green} branch has no errors when building and testing our client artifacts. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 11s{color} |
{color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  0m 44s{color} | {color:blue}
Used deprecated FindBugs config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 25s{color} |
{color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 26s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 29s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 48s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red}  1m 48s{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client generated 34 new + 95 unchanged - 34 fixed
= 129 total (was 129) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m 48s{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client generated 1 new + 336 unchanged - 0 fixed
= 337 total (was 336) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  0m 49s{color}
| {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 37
new + 594 unchanged - 0 fixed = 631 total (was 594) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 23s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 39s{color}
| {color:green} patch has no errors when building and testing our client artifacts. {color}
|
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 25s{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common generated
4 new + 96 unchanged - 4 fixed = 100 total (was 100) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  2s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app generated 2 new
+ 0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  6m 46s{color} | {color:green}
hadoop-mapreduce-client-core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  2s{color} | {color:green}
hadoop-mapreduce-client-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m  1s{color} | {color:green}
hadoop-mapreduce-client-app in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 12s{color} | {color:green}
hadoop-mapreduce-client-hs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}127m 45s{color} | {color:green}
hadoop-mapreduce-client-jobclient in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 49s{color} | {color:red}
The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}226m 20s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
|
|  |  Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.maxRunningMaps;
locked 57% of time  Unsynchronized access at RMContainerAllocator.java:57% of time  Unsynchronized
access at RMContainerAllocator.java:[line 964] |
|  |  Inconsistent synchronization of org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.maxRunningReduces;
locked 57% of time  Unsynchronized access at RMContainerAllocator.java:57% of time  Unsynchronized
access at RMContainerAllocator.java:[line 230] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7786/artifact/out/Dockerfile
|
| JIRA Issue | MAPREDUCE-7274 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13004007/MAPREDUCE-7274.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient
findbugs checkstyle cc prototool |
| uname | Linux 3c325c13f281 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / f43a152b972 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| cc | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7786/artifact/out/diff-compile-cc-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
|
| javac | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7786/artifact/out/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7786/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7786/artifact/out/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7786/artifact/out/new-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.html
|
|  Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7786/testReport/
|
| asflicense | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7786/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 1525 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7786/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |


This message was automatically generated.



> Enable to limit running map and reduce tasks when job is running
> ----------------------------------------------------------------
>
>                 Key: MAPREDUCE-7274
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7274
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, mrv2
>            Reporter: Chengwei Wang
>            Priority: Major
>         Attachments: MAPREDUCE-7274.patch
>
>
> MRv2 enabled users to control the number of map or reduce tasks running simultaneously
by configuration  *_mapreduce.job.running.map.limit_* or _*mapreduce.job.running.reduce.limit*._
But users can only set limit number before submitting the job to rm. So, it's meaningful 
to enable users to set the limit of running map or reduce tasks when job is running, which
can help users to restrict  resource usage of job and give resources to high-priority job.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message