hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6590) Update MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV value for loading correct navite lib when using ubertask feature
Date Sun, 27 Dec 2015 05:49:50 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15072055#comment-15072055
] 

Hadoop QA commented on MAPREDUCE-6590:
--------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 18s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 33s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 37s {color} |
{color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s {color} | {color:green}
trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color}
| {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 16s {color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has
2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s {color} |
{color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 55s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 6s {color} | {color:green}
the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 6s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 15s {color} |
{color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 15s {color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s {color} | {color:red}
Patch generated 1 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client
(total was 548, now 548). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 59s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s {color} |
{color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s {color} |
{color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 2s {color} | {color:green}
hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 101m 29s {color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 29s {color} | {color:green}
hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 19s {color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 32s {color} | {color:red}
Patch generated 18 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 251m 11s {color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.mapred.TestMiniMRChildTask |
|   | hadoop.mapred.TestNetworkedJob |
|   | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
| JDK v1.7.0_91 Failed junit tests | hadoop.mapred.TestMiniMRChildTask |
|   | hadoop.mapred.TestNetworkedJob |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12779559/MAPREDUCE-6590.patch
|
| JIRA Issue | MAPREDUCE-6590 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 67fe2de2a337 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8cfd672 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
|
| findbugs | v3.0.0 |
| findbugs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6239/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
|
| checkstyle | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6239/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
|
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6239/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
|
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6239/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
|
| unit test logs |  https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6239/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6239/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
|
| JDK v1.7.0_91  Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6239/testReport/
|
| asflicense | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6239/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
|
| Max memory used | 76MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT   http://yetus.apache.org |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6239/console |


This message was automatically generated.



> Update MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV value for loading correct navite lib
when using ubertask feature
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6590
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6590
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.7.1
>            Reporter: NING DING
>            Assignee: NING DING
>         Attachments: MAPREDUCE-6590.patch
>
>
> By default, the MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured
to set LD_LIBRARY_PATH, MR AM will fail to load the native library, then you can find the
error message as below.
> {panel}
> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable.
> {panel}
> I found the patch in https://issues.apache.org/jira/browse/MAPREDUCE-5799 update YARNRunner
code and use MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV as the default value. The code is as
below.
> {code:title=YARNRunner.java|borderStyle=solid}
>     MRApps.setEnvFromInputString(environment, 
>         conf.get(MRJobConfig.MR_AM_ADMIN_USER_ENV,
>             MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV), conf);
> {code}
> In fact the parameters yarn.app.mapreduce.am.env and yarn.app.mapreduce.am.admin.user.env
are resolved in YARNRunner running on hadoop client host.
> But their values, e.g. LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, will be used on
hadoop cluster hosts.
> If the hadoop client host installs hadoop on a different path from that on hadoop cluster
hosts,
> the $HADOOP_COMMON_HOME/lib/native will be resolved to a path which is not existed on
hadoop cluster hosts.
> Then MRAppMaster running on hadoop cluster host cannot load native lib.
> If don't use MAPREDUCE-5799 patch, I set my hadoop client's mapred-site.xml with the
following content.
> {code:xml}
> <property>
>         <name>yarn.app.mapreduce.am.admin.user.env</name>
>         <value>LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native</value>
       
> </property>
> {code}
> In this way, the YARNRunner will put LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native
into environment. 
> The method ContainerLaunch.expandEnvironment running in NodeManager can translate LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native

> to LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native. 
> The host running NodeManager can find its $HADOOP_COMMON_HOME/lib/native path.
> I suggest MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV should be defined as below.
> {code:title=MRJobConfig.java|borderStyle=solid}
> public final String DEFAULT_MAPRED_ADMIN_USER_ENV = 
>       Shell.WINDOWS ? 
>           "PATH={{PATH}};{{HADOOP_COMMON_HOME}}\\bin":
>           "LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native";
> {code}
> Please see my patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message