hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "genericqa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8569) Create an interface to provide cluster information to application
Date Fri, 31 Aug 2018 22:13:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599318#comment-16599318
] 

genericqa commented on YARN-8569:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 20s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 4 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 26s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m  0s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  4s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 21s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 41s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 19s{color}
| {color:green} branch has no errors when building and testing our client artifacts. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 18s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 46s{color} |
{color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 13s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 20s{color} | {color:red}
hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  1m 11s{color} | {color:red}
hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red}  1m 11s{color} | {color:red}
hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m 11s{color} | {color:red}
hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  1m  8s{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 28 new + 130 unchanged
- 1 fixed = 158 total (was 131) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 22s{color} | {color:red}
hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red}  3m 50s{color} | {color:red}
patch has errors when building and testing our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 23s{color} | {color:red}
hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 22s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated
1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 38s{color} | {color:green}
hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 23s{color} | {color:red}
hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 53s{color} | {color:red}
hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 14s{color} | {color:green}
hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 27s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 42s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.service.provider.TestAbstractProviderService |
|   | hadoop.yarn.service.component.TestComponent |
|   | hadoop.yarn.service.TestServiceAM |
|   | hadoop.yarn.service.utils.TestFilterUtils |
|   | hadoop.yarn.service.TestServiceManager |
|   | hadoop.yarn.service.client.TestServiceClient |
|   | hadoop.yarn.service.component.instance.TestComponentInstance |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8569 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12937957/YARN-8569.005.patch
|
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit
 shadedclient  findbugs  checkstyle  cc  |
| uname | Linux 527763dc86fc 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8aa6c4f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| mvninstall | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| compile | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
|
| cc | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
|
| javac | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/21731/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/21731/testReport/ |
| Max. process+thread count | 669 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/21731/console |
| Powered by | Apache Yetus 0.9.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Create an interface to provide cluster information to application
> -----------------------------------------------------------------
>
>                 Key: YARN-8569
>                 URL: https://issues.apache.org/jira/browse/YARN-8569
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Major
>              Labels: Docker
>         Attachments: YARN-8569 YARN sysfs interface to provide cluster information to
application.pdf, YARN-8569.001.patch, YARN-8569.002.patch, YARN-8569.003.patch, YARN-8569.004.patch,
YARN-8569.005.patch
>
>
> Some program requires container hostnames to be known for application to run.  For example,
distributed tensorflow requires launch_command that looks like:
> {code}
> # On ps0.example.com:
> $ python trainer.py \
>      --ps_hosts=ps0.example.com:2222,ps1.example.com:2222 \
>      --worker_hosts=worker0.example.com:2222,worker1.example.com:2222 \
>      --job_name=ps --task_index=0
> # On ps1.example.com:
> $ python trainer.py \
>      --ps_hosts=ps0.example.com:2222,ps1.example.com:2222 \
>      --worker_hosts=worker0.example.com:2222,worker1.example.com:2222 \
>      --job_name=ps --task_index=1
> # On worker0.example.com:
> $ python trainer.py \
>      --ps_hosts=ps0.example.com:2222,ps1.example.com:2222 \
>      --worker_hosts=worker0.example.com:2222,worker1.example.com:2222 \
>      --job_name=worker --task_index=0
> # On worker1.example.com:
> $ python trainer.py \
>      --ps_hosts=ps0.example.com:2222,ps1.example.com:2222 \
>      --worker_hosts=worker0.example.com:2222,worker1.example.com:2222 \
>      --job_name=worker --task_index=1
> {code}
> This is a bit cumbersome to orchestrate via Distributed Shell, or YARN services launch_command.
 In addition, the dynamic parameters do not work with YARN flex command.  This is the classic
pain point for application developer attempt to automate system environment settings as parameter
to end user application.
> It would be great if YARN Docker integration can provide a simple option to expose hostnames
of the yarn service via a mounted file.  The file content gets updated when flex command is
performed.  This allows application developer to consume system environment settings via a
standard interface.  It is like /proc/devices for Linux, but for Hadoop.  This may involve
updating a file in distributed cache, and allow mounting of the file via container-executor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message