hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5045) hbase unit tests fail due to dependency issues
Date Thu, 05 May 2016 22:17:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-5045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273218#comment-15273218
] 

Sangjin Lee commented on YARN-5045:
-----------------------------------

This is basically a *diamond dependency problem*. Timeline service depends on HBase 1.0.1
which in turn depends on hadoop-common *2.5.1*. However, timeline service itself depends on
hadoop-common *trunk*. In maven, the trunk version is chosen as the hadoop dependency version
is managed.

We knew there is potential risk of this diamond problem when the timeline service is developed
on trunk (where the backward incompatible changes can happen) while we also depend on HBase
which relies on a released version of hadoop. This is a first manifestation of the issue.

There are several ways we can address this issue, but there are drawbacks on all of the approaches.

(1) we can try to resurrect the metrics v.1 classes (that were removed in HADOOP-12504)
We could add back those classes under {{src/test/java}} of the timelineservice project. While
this might be an easy way to get around this specific problem, there are issues. First, it
is in poor taste to resurrect code we're getting rid of to work around this problem. Also,
it does nothing to handle potential future issues that may arise. What if classes' shapes
change instead of being removed?

(2) isolate the hbase unit test code and (en)force the hadoop version that HBase relies on
We could try to override the default version of hadoop-common (trunk: 3.0.0-SNAPSHOT) while
running the unit tests. It turns out this requires creating a separate project. Prior to maven
3, there used to be a capability of choosing a different version for the test scope and test
scope only. However, maven 3 or later seems to have removed that. Thus, for a given project,
there can only be one dependency version regardless of scope.

We could try to isolate the HBase-related unit tests into their own project and enforce 2.5.1
in that project. This has a decent chance of success.

Still there are caveats. For this to work, the portion of the timeline service code that exercises
HBase has to ensure it works fine against hadoop-common 2.5.1 as well as trunk. I'm not sure
how best to enforce this state.

Also, I thought of moving the non-test code that relies on HBase into this new project as
well, but I'm moving away from that idea. The main problem is that we need to prevent the
hadoop-common 2.5.1 jar from being incorporated into the hadoop distribution tarball. If we
have the non-test code in here, the worry is hadoop-common 2.5.1 may sneak into the tarball.

(3) pull out timeline service entirely out of the hadoop project
This is truly an out-of-the-box idea. Technically it might be feasible to set up the HBase-related
part of the timeline service as a separate project from hadoop. Then the hadoop-proper part
of the timeline service would interact with the HBase-related part of the timeline service
over the wire. That way, hadoop can still be shielded from the thorny HBase dependency issues.

While there is some possibility this may work technically, I think there are several major
issues, ranging from logistics of setting up a separate project independent of hadoop, performance
implications, etc. I think this is a long shot at best.

Out of these options, it seems to me (2) is the least bad option. I am working on making that
work. I'd love to hear your thoughts if you have better suggestions.

> hbase unit tests fail due to dependency issues
> ----------------------------------------------
>
>                 Key: YARN-5045
>                 URL: https://issues.apache.org/jira/browse/YARN-5045
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>            Priority: Blocker
>
> After the 5/4 rebase, the hbase unit tests in the timeline service project are failing:
> {noformat}
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 Time elapsed: 5.103 sec  <<< ERROR!
> java.io.IOException: Shutting down
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> 	at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> 	at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> 	at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
> 	at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
> 	at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> 	at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
> 	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> 	at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> 	at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:217)
> 	at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:153)
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:213)
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:978)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:938)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:750)
> 	at org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:87)
> {noformat}
> The root cause is that the hbase mini server depends on hadoop common's {{MetricsServlet}}
which has been removed in the trunk (HADOOP-12504):
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/metrics/MetricsServlet
>         at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
>         at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
>         at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
>         at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
>         at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
>         at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
>         at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
>         at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
>         ... 26 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message