hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4958) The file localization process should allow for wildcards to reduce the application footprint in the state store
Date Thu, 12 May 2016 21:38:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-4958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282095#comment-15282095
] 

Hadoop QA commented on YARN-4958:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 4 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s {color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 13s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 5s {color} | {color:green}
trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 2s {color} | {color:green}
trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 27s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 50s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 5s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 54s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 26s {color} |
{color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 36s {color} |
{color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 20s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 33s {color} |
{color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 7s {color} | {color:red}
root-jdk1.8.0_91 with JDK v1.8.0_91 generated 24 new + 671 unchanged - 0 fixed = 695 total
(was 671) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 33s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 2s {color} | {color:green}
the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 16m 9s {color} | {color:red}
root-jdk1.7.0_95 with JDK v1.7.0_95 generated 24 new + 680 unchanged - 0 fixed = 704 total
(was 680) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 2s {color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 26s {color} | {color:red}
root: patch generated 4 new + 233 unchanged - 16 fixed = 237 total (was 249) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 45s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 2s {color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red}
The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 44s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s {color} | {color:green}
the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s {color} |
{color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 14s {color} | {color:red}
hadoop-common in the patch failed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 9s {color} | {color:green}
hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 3s {color} | {color:green}
hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s {color} | {color:green}
hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 28s {color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 38s {color} | {color:green}
hadoop-common in the patch passed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 46s {color} | {color:green}
hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 26s {color} | {color:green}
hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s {color} | {color:green}
hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 107m 57s {color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 31s {color} | {color:red}
Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 327m 42s {color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed junit tests | hadoop.ha.TestZKFailoverController |
|   | hadoop.mapred.TestMiniMRChildTask |
| JDK v1.7.0_95 Failed junit tests | hadoop.mapred.TestMiniMRChildTask |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:cf2ee45 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12803663/YARN-4958.002.patch
|
| JIRA Issue | YARN-4958 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  xml  |
| uname | Linux 2f6028f3cbd3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3f2816a |
| Default Java | 1.7.0_95 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
|
| findbugs | v3.0.0 |
| javac | root-jdk1.8.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_91.txt
|
| javac | root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/diff-checkstyle-root.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/whitespace-eol.txt
|
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
|
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
|
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
|
| unit test logs |  https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
|
| JDK v1.7.0_95  Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/11428/testReport/
|
| asflicense | https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C:  hadoop-common-project/hadoop-common   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core   hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient  U: .
|
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/11428/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> The file localization process should allow for wildcards to reduce the application footprint
in the state store
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4958
>                 URL: https://issues.apache.org/jira/browse/YARN-4958
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.8.0
>            Reporter: Daniel Templeton
>            Assignee: Daniel Templeton
>            Priority: Critical
>         Attachments: YARN-4958.001.patch, YARN-4958.002.patch
>
>
> When using the -libjars option to add classes to the classpath, every library so added
is explicitly listed in the {{ContainerLaunchContext}}'s local resources even though they're
all uploaded to the same directory in HDFS.  When using tools like Crunch without an uber
JAR or when trying to take advantage of the shared cache, the number of libraries can be quite
large.  We've seen many cases where we had to turn down the max number of applications to
prevent ZK from running out of heap because of the size of the state store entries.
> Rather than listing all files independently, this JIRA proposes to have the NM allow
wildcards in the resource localization paths.  Specifically, we propose to allow a path to
have a final component (name) set to "*", which is interpreted by the NM as "download the
full directory and link to every file in it from the job's working directory."  This behavior
is the same as the current behavior when using -libjars, but avoids explicitly listing every
file.
> This JIRA does not attempt to provide more general purpose wildcards, such as "\*.jar"
or "file\*", as having multiple entries for a single directory presents numerous logistical
issues.
> This JIRA also does not attempt to integrate with the shared cache.  That work will be
left to a future JIRA.  Specifically, this JIRA only applies when a full directory is uploaded.
 Currently the shared cache does not handle directory uploads.
> This JIRA proposes to allow for wildcards both in the internal processing of the -libjars
switch and in paths added through the {{Job}} and {{DistributedCache}} classes.
> The proposed approach is to treat a path, "dir/\*", as "dir" for purposes of all file
verification and localization.  In the final step, the NM will query the localized directory
to get a list of the files in "dir" such that each can be linked from the job's working directory.
 Since $PWD/\* is always included on the classpath, all JAR files in "dir" will be in the
classpath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message