hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4882) Change the log level to DEBUG for recovering completed applications
Date Tue, 27 Dec 2016 23:10:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15781489#comment-15781489
] 

Hadoop QA commented on YARN-4882:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 13s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  0s{color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 46s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 35s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 25s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 36s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 16s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  6s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 26s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 31s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 30s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 30s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 20s{color}
| {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 252 unchanged - 2 fixed = 252 total (was 254) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 31s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 13s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  1s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 18s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 35s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 16s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 58s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-4882 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12844824/YARN-4882.004.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 2047ba30c913 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1bbd023 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/14480/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/14480/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/14480/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Change the log level to DEBUG for recovering completed applications
> -------------------------------------------------------------------
>
>                 Key: YARN-4882
>                 URL: https://issues.apache.org/jira/browse/YARN-4882
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>            Reporter: Rohith Sharma K S
>            Assignee: Daniel Templeton
>              Labels: oct16-easy
>         Attachments: YARN-4882.001.patch, YARN-4882.002.patch, YARN-4882.003.patch, YARN-4882.004.patch
>
>
> I think for recovering completed applications no need to log as INFO, rather it can be
made it as DEBUG.  The problem seen from large cluster is if any issue happens during RM start
up and continuously switching , then  RM logs are filled with most with recovering applications
only. 
> There are 6 lines are logged for 1 applications as I shown in below logs, then consider
RM default value for max-completed applications is 10K. So for each switch 10K*6=60K lines
will be added which is not useful I feel.
> {noformat}
> 2016-03-01 10:20:59,077 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager:
Default priority level is set to application:application_1456298208485_21507
> 2016-03-01 10:20:59,094 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
Recovering app: application_1456298208485_21507 with 1 attempts and final state = FINISHED
> 2016-03-01 10:20:59,100 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Recovering attempt: appattempt_1456298208485_21507_000001 with final state: FINISHED
> 2016-03-01 10:20:59,107 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1456298208485_21507_000001 State change from NEW to FINISHED
> 2016-03-01 10:20:59,111 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1456298208485_21507 State change from NEW to FINISHED
> 2016-03-01 10:20:59,112 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
USER=rohith   OPERATION=Application Finished - Succeeded      TARGET=RMAppManager     RESULT=SUCCESS
 APPID=application_1456298208485_21507
> {noformat}
> The main problem is missing important information's from the logs before RM unstable.
Even though log roll back is 50 or 100, in a short period all these logs will be rolled out
and all the logs contains only RM switching information that too recovering applications!!.

> I suggest at least completed applications recovery should be logged as DEBUG.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message