hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4709) Exception when option to fetch all log files is specified while using yarn logs -am command and incorrect JSON produced for containerLogFiles
Date Sun, 21 Feb 2016 18:42:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156153#comment-15156153
] 

Hadoop QA commented on YARN-4709:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 1s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 2 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s {color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 42s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s {color} |
{color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 3s {color} | {color:green}
trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 32s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 25s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 55s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s {color} |
{color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s {color} |
{color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s {color} |
{color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 39s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s {color} | {color:green}
the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 22s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 17s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s {color} |
{color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s {color} |
{color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s {color} | {color:green}
hadoop-yarn-common in the patch passed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 26s {color} | {color:red}
hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s {color} | {color:green}
hadoop-yarn-common in the patch passed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 55s {color} | {color:red}
hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 12s {color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesContainers
|
| JDK v1.7.0_95 Failed junit tests | hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesContainers
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12788894/YARN-4709.01.patch
|
| JIRA Issue | YARN-4709 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 56ad88cd1274 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d5abd29 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
|
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/10594/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_72.txt
|
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/10594/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt
|
| unit test logs |  https://builds.apache.org/job/PreCommit-YARN-Build/10594/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_72.txt
https://builds.apache.org/job/PreCommit-YARN-Build/10594/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt
|
| JDK v1.7.0_95  Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/10594/testReport/
|
| modules | C:  hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/10594/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Exception when option to fetch all log files is specified while using yarn logs -am command
and incorrect JSON produced for containerLogFiles
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4709
>                 URL: https://issues.apache.org/jira/browse/YARN-4709
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-4709.01.patch
>
>
> Following exception is thrown when we run below command.
> {panel}
> root@varun-Inspiron-5558:/opt1/hadoop3/bin# ./yarn logs -applicationId application_1455999168135_0002
-am ALL -logFiles ALL
> Container: container_e31_1455999168135_0002_01_000001
> =======================================================
> {color:red}LogType:syslogstderrstdout
> Log Upload Time:Sun Feb 21 01:44:55 +0530 2016
> Log Contents:
> java.lang.Exception: Cannot find this log on the local disk.
> End of LogType:syslogstderrstdout{color}
> LogType:syslog
> Log Upload Time:Sun Feb 21 01:44:55 +0530 2016
> Log Contents:
> 2016-02-21 01:44:49,565 INFO \[main\] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Created MRAppMaster for application appattempt_1455999168135_0002_000001
> 2016-02-21 01:44:49,914 INFO \[main\] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:

> /************************************************************
> {panel}
> This is because we annotate containerLogFiles list with XmlElementWrapper which generates
XML output as under. And when we read this XML at client side, reading the value associated
with containerLogFiles also leads to one value being syslogstderrstdout because both parent
and child tags are same. This leads to the exception. 
> {noformat}
> <containerLogFiles>
>   <containerLogFiles>syslog</containerLogFiles>
>   <containerLogFiles>stderr</containerLogFiles>
>   <containerLogFiles>stdout</containerLogFiles>
> </containerLogFiles>
> {noformat}
> Moreover, as we use XMLElementWrapper, the JSON generated is as under. This JSON cannot
be properly parsed by JSON parser(as a list). This is because child containerLogsFiles entries
are treated as a key-value pair(map) and hence only last entry i.e. stdout is picked up. This
was found while working on YARN-4517. This makes output unusable. 
> This will be an issue for 2 REST endpoints i.e. {{/ws/v1/node/containers}} and {{/ws/v1/node/containers/\{\{containerId\}\}}}
> {noformat}
>   "containerLogFiles":[
>     {
>       "containerLogFiles":"syslog",
>       "containerLogFiles":"stderr",
>       "containerLogFiles":"stdout"
>     }
>   ]
> {noformat}
> Ideally the JSON output should be as under.
> {noformat}
> "containerLogFiles":["syslog","stderr","stdout"]
> {noformat}
> We can indicate in the JAXB context to ignore the outer wrapper while marshalling to
JSON. But this can only be done at class level. If we do so for ContainerInfo, it would break
backward compatibility.
> Hence, to fix it we can remove XmlElementWrapper annotation for containerLogFiles list.
> Another solution would be to wrap the list inside another class.
> But going with former as of now as we do not specify XmlElementWrapper for lists at most
of the places in our code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message