hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-888) clean up POM dependencies
Date Thu, 09 Jan 2014 17:50:55 GMT

    [ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13866845#comment-13866845
] 

Alejandro Abdelnur edited comment on YARN-888 at 1/9/14 5:50 PM:
-----------------------------------------------------------------

[~vinodkv], 

While having dependencies in non-leaf POMs may reduce the size of leaf POMs, it drags non-required
dependencies (unless you only put in non-leaf POMs dependencies that are common to all the
leaf modules).

Yes IntelliJ seems to get funny with dependencies in non-leaf modules. That is one of the
motivations (agree it is an IntelliJ issue, on the other hand the change does not affect the
project built at all and allows IntelliJ users to build/debug from the IDE out of the box
without doing funny voodoo).

The other motivation, and IMO is more important, is to clean up the dependencies modules like
yarn-api and yarn-client have. Restricting them to what is used on the client side. Using
the dependency:tree and dependency:analyze plugins I’ve reduced the 3rd party JARs required
by the clients significantly. As [~stevel@apache.org] pointed out there is much more work
we should do in this direction, this is a first non-intrusive baby step in that direction.

To give you and idea, before the this patch *hadoop-yarn-api* reports as required dependencies
by itself:

{code}
 +- org.slf4j:slf4j-api:jar:1.7.5:compile
 +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
 |  \- log4j:log4j:jar:1.2.17:compile
 +- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
 |  +- tomcat:jasper-compiler:jar:5.5.23:test
 +- com.google.inject.extensions:guice-servlet:jar:3.0:compile
 +- io.netty:netty:jar:3.6.2.Final:compile
 +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
 +- commons-io:commons-io:jar:2.4:compile
 +- com.google.inject:guice:jar:3.0:compile
 |  +- javax.inject:javax.inject:jar:1:compile
 |  \- aopalliance:aopalliance:jar:1.0:compile
 +- com.sun.jersey:jersey-server:jar:1.9:compile
 |  +- asm:asm:jar:3.2:compile
 |  \- com.sun.jersey:jersey-core:jar:1.9:compile
 +- com.sun.jersey:jersey-json:jar:1.9:compile
 |  +- org.codehaus.jettison:jettison:jar:1.1:compile
 |  |  \- stax:stax-api:jar:1.0.1:compile
 |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
 |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
 |  |     \- javax.activation:activation:jar:1.1:compile
 |  +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
 |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
 |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.8:compile (version managed from 1.8.3)
 |  \- org.codehaus.jackson:jackson-xc:jar:1.8.8:compile (version managed from 1.8.3)
 \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
{code}

With the patch, the required dependencies by itself are down to:

{code}
 +- commons-lang:commons-lang:jar:2.6:compile
 +- com.google.guava:guava:jar:11.0.2:compile
 |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
 +- commons-logging:commons-logging:jar:1.1.3:compile
 +- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
 \- com.google.protobuf:protobuf-java:jar:2.5.0:compile
{code}

Does this address your concerns?


was (Author: tucu00):
[~vinodvk], 

While having dependencies in non-leaf POMs may reduce the size of leaf POMs, it drags non-required
dependencies (unless you only put in non-leaf POMs dependencies that are common to all the
leaf modules).

Yes IntelliJ seems to get funny with dependencies in non-leaf modules. That is one of the
motivations (agree it is an IntelliJ issue, on the other hand the change does not affect the
project built at all and allows IntelliJ users to build/debug from the IDE out of the box
without doing funny voodoo).

The other motivation, and IMO is more important, is to clean up the dependencies modules like
yarn-api and yarn-client have. Restricting them to what is used on the client side. Using
the dependency:tree and dependency:analyze plugins I’ve reduced the 3rd party JARs required
by the clients significantly. As [~stevel@apache.org] pointed out there is much more work
we should do in this direction, this is a first non-intrusive baby step in that direction.

To give you and idea, before the this patch *hadoop-yarn-api* reports as required dependencies
by itself:

{code}
 +- org.slf4j:slf4j-api:jar:1.7.5:compile
 +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
 |  \- log4j:log4j:jar:1.2.17:compile
 +- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
 |  +- tomcat:jasper-compiler:jar:5.5.23:test
 +- com.google.inject.extensions:guice-servlet:jar:3.0:compile
 +- io.netty:netty:jar:3.6.2.Final:compile
 +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
 +- commons-io:commons-io:jar:2.4:compile
 +- com.google.inject:guice:jar:3.0:compile
 |  +- javax.inject:javax.inject:jar:1:compile
 |  \- aopalliance:aopalliance:jar:1.0:compile
 +- com.sun.jersey:jersey-server:jar:1.9:compile
 |  +- asm:asm:jar:3.2:compile
 |  \- com.sun.jersey:jersey-core:jar:1.9:compile
 +- com.sun.jersey:jersey-json:jar:1.9:compile
 |  +- org.codehaus.jettison:jettison:jar:1.1:compile
 |  |  \- stax:stax-api:jar:1.0.1:compile
 |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
 |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
 |  |     \- javax.activation:activation:jar:1.1:compile
 |  +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
 |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
 |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.8:compile (version managed from 1.8.3)
 |  \- org.codehaus.jackson:jackson-xc:jar:1.8.8:compile (version managed from 1.8.3)
 \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
{code}

With the patch, the required dependencies by itself are down to:

{code}
 +- commons-lang:commons-lang:jar:2.6:compile
 +- com.google.guava:guava:jar:11.0.2:compile
 |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
 +- commons-logging:commons-logging:jar:1.1.3:compile
 +- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
 \- com.google.protobuf:protobuf-java:jar:2.5.0:compile
{code}

Does this address your concerns?

> clean up POM dependencies
> -------------------------
>
>                 Key: YARN-888
>                 URL: https://issues.apache.org/jira/browse/YARN-888
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.1.0-beta
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: YARN-888.patch, YARN-888.patch, yarn-888-2.patch
>
>
> Intermediate 'pom' modules define dependencies inherited by leaf modules.
> This is causing issues in intellij IDE.
> We should normalize the leaf modules like in common, hdfs and tools where all dependencies
are defined in each leaf module and the intermediate 'pom' module do not define any dependency.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message