hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5045) hbase unit tests fail due to dependency issues
Date Fri, 06 May 2016 12:14:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-5045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273960#comment-15273960
] 

Varun Saxena commented on YARN-5045:
------------------------------------

[~sjlee0], thanks for the patch.
The patch looks fine. I ran the test cases and they all passed.

One question though. When I printed dependency tree for timelineservice hbase tests project,
I still see some 3.0.0-SNAPSHOT jars being taken. Few anyways should be taken, such as hadoop-yarn-api,
hadoop-yarn-common, etc.
But some seem to be pulled unnecessarily/wrongly. For instance, hadoop-common is pulling in
hadoop-auth 3.0.0-SNAPSHOT jar. Shouldnt this be 2.5.1 too ? To avoid later surprises.
We can set it to 2.5.1 and exclude wherever 3.0.0-SNAPSHOT is being pulled in. However, it
is possible hadoop-auth may not be used at runtime.

Also, I see some mapreduce 3.0.0-SNAPSHOT jars being pulled via hbase dependencies. I am not
sure how mapreduce jars will be useful in our test flow at runtime. Anyways should we enforce
2.5.1 version on them too. Or exclude them ?
I did try excluding Mapreduce jars and setting hadoop-auth to 2.5.1. Tests passed even after
these changes. 

{noformat}
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop YARN Timeline Service HBase tests 3.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.2:tree (default-cli) @ hadoop-yarn-server-timelineservice-hbase-tests
---
[INFO] org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-tests:jar:3.0.0-SNAPSHOT
[INFO] +- junit:junit:jar:4.11:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.apache.hadoop:hadoop-yarn-server-timelineservice:jar:3.0.0-SNAPSHOT:test
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:test
[INFO] |  |  \- jdk.tools:jdk.tools:jar:1.7:system
[INFO] |  +- com.google.inject:guice:jar:3.0:test
[INFO] |  |  +- javax.inject:javax.inject:jar:1:test
[INFO] |  |  \- aopalliance:aopalliance:jar:1.0:test
[INFO] |  +- javax.servlet:servlet-api:jar:2.5:test
[INFO] |  +- javax.xml.bind:jaxb-api:jar:2.2.2:test
[INFO] |  |  +- javax.xml.stream:stax-api:jar:1.0-2:test
[INFO] |  |  \- javax.activation:activation:jar:1.1:test
[INFO] |  +- commons-cli:commons-cli:jar:1.2:test
[INFO] |  +- commons-lang:commons-lang:jar:2.6:test
[INFO] |  +- commons-logging:commons-logging:jar:1.1.3:test
[INFO] |  +- org.apache.commons:commons-csv:jar:1.0:test
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:test
[INFO] |  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:test
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.5.1:test
[INFO] |  +- org.apache.commons:commons-math3:jar:3.1.1:test
[INFO] |  +- xmlenc:xmlenc:jar:0.52:test
[INFO] |  +- commons-httpclient:commons-httpclient:jar:3.1:test
[INFO] |  +- commons-codec:commons-codec:jar:1.4:test
[INFO] |  +- commons-net:commons-net:jar:3.1:test
[INFO] |  +- commons-collections:commons-collections:jar:3.2.2:test
[INFO] |  +- org.mortbay.jetty:jetty:jar:6.1.26:test
[INFO] |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:test
[INFO] |  +- com.sun.jersey:jersey-json:jar:1.9:test
[INFO] |  |  \- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:test
[INFO] |  +- com.sun.jersey:jersey-server:jar:1.9:test
[INFO] |  |  \- asm:asm:jar:3.2:test (version managed from 3.1)
[INFO] |  +- tomcat:jasper-compiler:jar:5.5.23:test
[INFO] |  +- tomcat:jasper-runtime:jar:5.5.23:test
[INFO] |  +- javax.servlet.jsp:jsp-api:jar:2.1:test
[INFO] |  +- commons-el:commons-el:jar:1.0:test
[INFO] |  +- log4j:log4j:jar:1.2.17:test
[INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.9.0:test
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.2.5:test
[INFO] |  |  \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:test
[INFO] |  +- commons-configuration:commons-configuration:jar:1.6:test
[INFO] |  |  +- commons-digester:commons-digester:jar:1.8:test
[INFO] |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:test
[INFO] |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:test
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.10:test
[INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.7.10:test
[INFO] |  +- org.apache.avro:avro:jar:1.7.4:test
[INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:test
[INFO] |  |  \- org.xerial.snappy:snappy-java:jar:1.0.4.1:test
[INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:test
[INFO] |  +- org.apache.hadoop:hadoop-auth:jar:3.0.0-SNAPSHOT:test
[INFO] |  |  +- com.nimbusds:nimbus-jose-jwt:jar:3.9:compile (scope managed from test)
[INFO] |  |  |  +- net.jcip:jcip-annotations:jar:1.0:compile
[INFO] |  |  |  +- net.minidev:json-smart:jar:1.1.1:compile
[INFO] |  |  |  \- commons-io:commons-io:jar:2.4:compile
[INFO] |  |  +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:test
[INFO] |  |  |  +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:test
[INFO] |  |  |  +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:test
[INFO] |  |  |  \- org.apache.directory.api:api-util:jar:1.0.0-M20:test
[INFO] |  |  \- org.apache.curator:curator-framework:jar:2.7.1:test
[INFO] |  |     \- org.apache.curator:curator-client:jar:2.7.1:test
[INFO] |  +- com.jcraft:jsch:jar:0.1.51:test (version managed from 0.1.42)
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.0:test
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:test
[INFO] |  \- org.apache.commons:commons-compress:jar:1.4.1:test
[INFO] |     \- org.tukaani:xz:jar:1.0:test
[INFO] +- org.apache.hadoop:hadoop-yarn-api:jar:3.0.0-SNAPSHOT:test
[INFO] +- org.apache.hadoop:hadoop-yarn-common:jar:3.0.0-SNAPSHOT:test
[INFO] |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:test (version managed from 1.8.3)
[INFO] |  +- org.codehaus.jackson:jackson-xc:jar:1.9.13:test (version managed from 1.8.3)
[INFO] |  +- com.google.inject.extensions:guice-servlet:jar:3.0:test
[INFO] |  \- com.sun.jersey.contribs:jersey-guice:jar:1.9:test
[INFO] +- org.apache.hadoop:hadoop-yarn-server-common:jar:3.0.0-SNAPSHOT:test
[INFO] |  \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:test
[INFO] +- org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:3.0.0-SNAPSHOT:test
[INFO] |  +- org.codehaus.jettison:jettison:jar:1.1:test
[INFO] |  \- de.ruedigermoeller:fst:jar:2.24:test
[INFO] |     +- org.javassist:javassist:jar:3.18.1-GA:test
[INFO] |     \- org.objenesis:objenesis:jar:2.1:test
[INFO] +- com.google.guava:guava:jar:11.0.2:test
[INFO] +- com.sun.jersey:jersey-core:jar:1.9:test
[INFO] +- com.sun.jersey:jersey-client:jar:1.9:test
[INFO] +- org.apache.hbase:hbase-common:jar:1.0.1:test
[INFO] |  +- org.apache.hbase:hbase-annotations:jar:1.0.1:test
[INFO] |  +- org.apache.htrace:htrace-core:jar:3.1.0-incubating:test
[INFO] |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.0.0-SNAPSHOT:test (version
managed from 2.5.1)
[INFO] |  \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:test
[INFO] +- org.apache.hbase:hbase-client:jar:1.0.1:test
[INFO] |  +- org.apache.hbase:hbase-protocol:jar:1.0.1:test
[INFO] |  +- io.netty:netty-all:jar:4.1.0.Beta5:test (version managed from 4.0.23.Final)
[INFO] |  +- org.jruby.jcodings:jcodings:jar:1.0.8:test
[INFO] |  \- org.jruby.joni:joni:jar:2.1.2:test
[INFO] +- org.apache.hbase:hbase-server:jar:1.0.1:test
[INFO] |  +- org.apache.hbase:hbase-prefix-tree:jar:1.0.1:test
[INFO] |  +- org.apache.hbase:hbase-hadoop-compat:jar:1.0.1:test
[INFO] |  +- org.apache.hbase:hbase-hadoop2-compat:jar:1.0.1:test
[INFO] |  +- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO] |  +- org.apache.commons:commons-math:jar:2.2:test
[INFO] |  +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:test
[INFO] |  +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:test
[INFO] |  +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:test
[INFO] |  +- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:test
[INFO] |  +- org.jamon:jamon-runtime:jar:2.3.1:test
[INFO] |  +- com.lmax:disruptor:jar:3.3.0:test
[INFO] |  \- org.apache.hadoop:hadoop-client:jar:3.0.0-SNAPSHOT:test (version managed from
2.5.1)
[INFO] |     \- org.apache.hadoop:hadoop-mapreduce-client-app:jar:3.0.0-SNAPSHOT:test
[INFO] +- org.apache.hbase:hbase-common:jar:tests:1.0.1:test
[INFO] +- org.apache.hbase:hbase-server:jar:tests:1.0.1:test
[INFO] +- org.apache.hbase:hbase-it:jar:tests:1.0.1:test
[INFO] |  +- org.apache.hbase:hbase-shell:jar:1.0.1:test
[INFO] |  |  \- org.jruby:jruby-complete:jar:1.6.8:test
[INFO] |  \- org.jboss.netty:netty:jar:3.2.4.Final:test
[INFO] +- org.apache.phoenix:phoenix-core:jar:4.5.0-SNAPSHOT:test
[INFO] |  +- org.antlr:antlr:jar:3.5:test
[INFO] |  |  \- org.antlr:ST4:jar:4.0.7:test
[INFO] |  +- org.antlr:antlr-runtime:jar:3.5:test
[INFO] |  |  \- org.antlr:stringtemplate:jar:3.2.1:test
[INFO] |  |     \- antlr:antlr:jar:2.7.7:test
[INFO] |  +- sqlline:sqlline:jar:1.1.8:test
[INFO] |  +- net.sourceforge.findbugs:annotations:jar:1.3.2:test
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.2.5:test
[INFO] |  +- org.iq80.snappy:snappy:jar:0.3:test
[INFO] |  \- org.apache.hbase:hbase-server:test-jar:tests:1.0.1:test
[INFO] +- org.apache.phoenix:phoenix-core:test-jar:tests:4.5.0-SNAPSHOT:test
[INFO] |  +- jline:jline:jar:2.11:test
[INFO] |  \- joda-time:joda-time:jar:2.7:test
[INFO] +- org.mockito:mockito-all:jar:1.8.5:test
[INFO] +- org.apache.hadoop:hadoop-common:test-jar:tests:2.5.1:test
[INFO] +- org.apache.hadoop:hadoop-hdfs:jar:2.5.1:test
[INFO] |  +- commons-daemon:commons-daemon:jar:1.0.13:test
[INFO] |  \- io.netty:netty:jar:3.6.2.Final:test
[INFO] +- org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.5.1:test
[INFO] \- org.apache.hbase:hbase-testing-util:jar:1.0.1:test
[INFO]    +- org.apache.hbase:hbase-common:test-jar:tests:1.0.1:test
[INFO]    +- org.apache.hbase:hbase-annotations:test-jar:tests:1.0.1:test
[INFO]    +- org.apache.hbase:hbase-hadoop-compat:test-jar:tests:1.0.1:test
[INFO]    +- org.apache.hbase:hbase-hadoop2-compat:test-jar:tests:1.0.1:test
[INFO]    \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:3.0.0-SNAPSHOT:test
[INFO]       +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:3.0.0-SNAPSHOT:test
[INFO]       |  \- org.apache.hadoop:hadoop-yarn-client:jar:3.0.0-SNAPSHOT:test
[INFO]       \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:3.0.0-SNAPSHOT:test
[INFO]          \- org.apache.hadoop:hadoop-yarn-server-nodemanager:jar:3.0.0-SNAPSHOT:test
[INFO]             \- com.codahale.metrics:metrics-core:jar:3.0.1:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.132s
[INFO] Finished at: Fri May 06 15:55:11 IST 2016
[INFO] Final Memory: 33M/689M
[INFO] ------------------------------------------------------------------------
{noformat}

> hbase unit tests fail due to dependency issues
> ----------------------------------------------
>
>                 Key: YARN-5045
>                 URL: https://issues.apache.org/jira/browse/YARN-5045
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>            Priority: Blocker
>         Attachments: YARN-5045-YARN-2928.01.patch, YARN-5045-YARN-2928.poc.patch
>
>
> After the 5/4 rebase, the hbase unit tests in the timeline service project are failing:
> {noformat}
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 Time elapsed: 5.103 sec  <<< ERROR!
> java.io.IOException: Shutting down
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> 	at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> 	at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> 	at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
> 	at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
> 	at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> 	at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
> 	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> 	at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> 	at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:217)
> 	at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:153)
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:213)
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:978)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:938)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:750)
> 	at org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:87)
> {noformat}
> The root cause is that the hbase mini server depends on hadoop common's {{MetricsServlet}}
which has been removed in the trunk (HADOOP-12504):
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/metrics/MetricsServlet
>         at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
>         at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
>         at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
>         at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
>         at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
>         at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
>         at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
>         at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
>         ... 26 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message