hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14685) Test jars to exclude from hadoop-client-minicluster jar
Date Fri, 28 Jul 2017 02:12:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104329#comment-16104329
] 

Sean Busbey commented on HADOOP-14685:
--------------------------------------

the test-jar artifacts aren't the problem. We probably need to include them, as that's where
the minicluster classes are defined. (and those are why we create this artifact in the first
place).

What we don't need are these things that got flagged on HADOOP-14089:

{code}

testjar/
testjar/ClassWordCount$MapClass.class
testjar/ClassWordCount$Reduce.class
testjar/ClassWordCount.class
testjar/CustomOutputCommitter.class
testjar/ExternalIdentityReducer.class
testjar/ExternalMapperReducer.class
testjar/ExternalWritable.class
testjar/JobKillCommitter$CommitterWithFailCleanup.class
testjar/JobKillCommitter$CommitterWithFailSetup.class
testjar/JobKillCommitter$CommitterWithNoError.class
testjar/JobKillCommitter$MapperFail.class
testjar/JobKillCommitter$MapperPass.class
testjar/JobKillCommitter$MapperPassSleep.class
testjar/JobKillCommitter$ReducerFail.class
testjar/JobKillCommitter$ReducerPass.class
testjar/JobKillCommitter.class
testjar/UserNamePermission$UserNameMapper.class
testjar/UserNamePermission$UserNameReducer.class
testjar/UserNamePermission.class
testshell/
testshell/ExternalMapReduce$MapClass.class
testshell/ExternalMapReduce$Reduce.class
testshell/ExternalMapReduce.class
.options
jdtCompilerAdapter.jar
{code}

I haven't chased down where these come from, but we should filter out just them.

We probably also ought to exclude unneeded properties files that come out of the test jars
too, most importantly the log4j.properties file since that should be up to downstream.

> Test jars to exclude from hadoop-client-minicluster jar
> -------------------------------------------------------
>
>                 Key: HADOOP-14685
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14685
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: common
>    Affects Versions: 3.0.0-beta1
>            Reporter: Bharat Viswanadham
>            Assignee: Bharat Viswanadham
>
> This jira is to discuss, what test jars to be included/excluded from hadoop-client-minicluster
> Jars included/excluded when building hadoop-client-minicluster
> [INFO] --- maven-shade-plugin:2.4.3:shade (default) @ hadoop-client-minicluster ---
> [INFO] Excluding org.apache.hadoop:hadoop-client-api:jar:3.0.0-beta1-SNAPSHOT from the
shaded jar.
> [INFO] Excluding org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-beta1-SNAPSHOT from
the shaded jar.
> [INFO] Excluding org.apache.htrace:htrace-core4:jar:4.1.0-incubating from the shaded
jar.
> [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.25 from the shaded jar.
> [INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar.
> [INFO] Excluding junit:junit:jar:4.11 from the shaded jar.
> [INFO] Including org.hamcrest:hamcrest-core:jar:1.3 in the shaded jar.
> [INFO] Excluding org.apache.hadoop:hadoop-annotations:jar:3.0.0-beta1-SNAPSHOT from the
shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-minicluster:jar:3.0.0-beta1-SNAPSHOT in the
shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-yarn-server-tests:test-jar:tests:3.0.0-beta1-SNAPSHOT
in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:3.0.0-beta1-SNAPSHOT
in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:3.0.0-beta1-SNAPSHOT
in the shaded jar.
> [INFO] Including de.ruedigermoeller:fst:jar:2.50 in the shaded jar.
> [INFO] Including com.cedarsoftware:java-util:jar:1.9.0 in the shaded jar.
> [INFO] Including com.cedarsoftware:json-io:jar:2.5.1 in the shaded jar.
> [INFO] Including org.apache.curator:curator-test:jar:2.12.0 in the shaded jar.
> [INFO] Including org.javassist:javassist:jar:3.18.1-GA in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-hdfs:jar:3.0.0-beta1-SNAPSHOT in the shaded
jar.
> [INFO] Including org.eclipse.jetty:jetty-util-ajax:jar:9.3.11.v20160721 in the shaded
jar.
> [INFO] Including commons-daemon:commons-daemon:jar:1.0.13 in the shaded jar.
> [INFO] Including io.netty:netty-all:jar:4.0.23.Final in the shaded jar.
> [INFO] Including xerces:xercesImpl:jar:2.9.1 in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-hs:jar:3.0.0-beta1-SNAPSHOT
in the shaded jar.
> [INFO] Excluding org.apache.hadoop:hadoop-yarn-server-timelineservice:jar:3.0.0-beta1-SNAPSHOT
from the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-common:test-jar:tests:3.0.0-beta1-SNAPSHOT
in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-hdfs:test-jar:tests:3.0.0-beta1-SNAPSHOT in
the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-jobclient:test-jar:tests:3.0.0-beta1-SNAPSHOT
in the shaded jar.
> [INFO] Including com.sun.jersey:jersey-core:jar:1.19 in the shaded jar.
> [INFO] Including com.sun.jersey:jersey-client:jar:1.19 in the shaded jar.
> [INFO] Including com.sun.jersey:jersey-json:jar:1.19 in the shaded jar.
> [INFO] Including org.codehaus.jettison:jettison:jar:1.1 in the shaded jar.
> [INFO] Including com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 in the shaded jar.
> [INFO] Including com.sun.jersey:jersey-server:jar:1.19 in the shaded jar.
> [INFO] Including com.sun.jersey:jersey-servlet:jar:1.19 in the shaded jar.
> [INFO] Including org.eclipse.jdt:core:jar:3.1.1 in the shaded jar.
> [INFO] Including net.sf.kosmosfs:kfs:jar:0.3 in the shaded jar.
> [INFO] Including net.java.dev.jets3t:jets3t:jar:0.9.0 in the shaded jar.
> [INFO] Including com.jamesmurty.utils:java-xmlbuilder:jar:0.4 in the shaded jar.
> [INFO] Including com.jcraft:jsch:jar:0.1.54 in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-yarn-server-nodemanager:jar:3.0.0-beta1-SNAPSHOT
in the shaded jar.
> [INFO] Including com.codahale.metrics:metrics-core:jar:3.0.1 in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:3.0.0-beta1-SNAPSHOT
in the shaded jar.
> [INFO] Including org.eclipse.jetty:jetty-server:jar:9.3.11.v20160721 in the shaded jar.
> [INFO] Including org.eclipse.jetty:jetty-http:jar:9.3.11.v20160721 in the shaded jar.
> [INFO] Including org.eclipse.jetty:jetty-io:jar:9.3.11.v20160721 in the shaded jar.
> [INFO] Including com.google.inject.extensions:guice-servlet:jar:4.0 in the shaded jar.
> [INFO] Including com.google.inject:guice:jar:4.0 in the shaded jar.
> [INFO] Including javax.inject:javax.inject:jar:1 in the shaded jar.
> [INFO] Including aopalliance:aopalliance:jar:1.0 in the shaded jar.
> [INFO] Including com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2:jar:1.19
in the shaded jar.
> [INFO] Including com.sun.jersey:jersey-grizzly2-servlet:jar:1.19 in the shaded jar.
> [INFO] Including org.glassfish.grizzly:grizzly-http-servlet:jar:2.2.21 in the shaded
jar.
> [INFO] Including org.glassfish.grizzly:grizzly-http-server:jar:2.2.21 in the shaded jar.
> [INFO] Including org.glassfish.grizzly:grizzly-rcm:jar:2.2.21 in the shaded jar.
> [INFO] Including com.sun.jersey:jersey-grizzly2:jar:1.19 in the shaded jar.
> [INFO] Including org.glassfish.grizzly:grizzly-http:jar:2.2.21 in the shaded jar.
> [INFO] Including org.glassfish.grizzly:grizzly-framework:jar:2.2.21 in the shaded jar.
> [INFO] Including com.sun.jersey.contribs:jersey-guice:jar:1.19 in the shaded jar.
> [INFO] Replacing original artifact with shaded artifact.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message