hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5045) hbase unit tests fail due to dependency issues
Date Fri, 06 May 2016 15:44:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-5045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274235#comment-15274235
] 

Sangjin Lee commented on YARN-5045:
-----------------------------------

Thanks for the comments [~varun_saxena]. I did notice those when I was working on the patch.

First the yarn dependencies should be fine. HBase does not depend on yarn, and including yarn
3.0.0 dependencies is orthogonal. What matters more is the hdfs/common dependencies as they
are shared between the timeline service unit tests and hbase.

We could try to enforce 2.5.1 on mapreduce job client, but this has a potential of making
the pom much bigger. Since all hadoop dependency versions are managed via dependency management
(hadoop-project/pom.xml), we need to exclude it first, and declare a new dependency. But we
may need to declare its dependencies too (as they are also managed, and without it 3.0.0 will
be used, etc.). Mapreduce-client is near the top of the dependency chain, and the number of
dependencies that needs to be replaced in this manner will be big. I haven't tried it, but
that's my suspicion.

My guess is that our unit tests are not exercising the mapreduce job client and that's probably
why there are no issues. Let me know what you think.

> hbase unit tests fail due to dependency issues
> ----------------------------------------------
>
>                 Key: YARN-5045
>                 URL: https://issues.apache.org/jira/browse/YARN-5045
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>            Priority: Blocker
>         Attachments: YARN-5045-YARN-2928.01.patch, YARN-5045-YARN-2928.poc.patch
>
>
> After the 5/4 rebase, the hbase unit tests in the timeline service project are failing:
> {noformat}
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 Time elapsed: 5.103 sec  <<< ERROR!
> java.io.IOException: Shutting down
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> 	at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> 	at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> 	at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
> 	at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
> 	at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> 	at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
> 	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> 	at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> 	at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:217)
> 	at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:153)
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:213)
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:978)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:938)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:750)
> 	at org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:87)
> {noformat}
> The root cause is that the hbase mini server depends on hadoop common's {{MetricsServlet}}
which has been removed in the trunk (HADOOP-12504):
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/metrics/MetricsServlet
>         at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
>         at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
>         at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
>         at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
>         at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
>         at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
>         at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
>         at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
>         ... 26 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message