Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C2E35200AC6 for ; Fri, 6 May 2016 17:44:14 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C19AA160A0C; Fri, 6 May 2016 15:44:14 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E8A741608F8 for ; Fri, 6 May 2016 17:44:13 +0200 (CEST) Received: (qmail 11463 invoked by uid 500); 6 May 2016 15:44:13 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 11439 invoked by uid 99); 6 May 2016 15:44:13 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2016 15:44:13 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id DD3942C1F64 for ; Fri, 6 May 2016 15:44:12 +0000 (UTC) Date: Fri, 6 May 2016 15:44:12 +0000 (UTC) From: "Sangjin Lee (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-5045) hbase unit tests fail due to dependency issues MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 06 May 2016 15:44:14 -0000 [ https://issues.apache.org/jira/browse/YARN-5045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274235#comment-15274235 ] Sangjin Lee commented on YARN-5045: ----------------------------------- Thanks for the comments [~varun_saxena]. I did notice those when I was working on the patch. First the yarn dependencies should be fine. HBase does not depend on yarn, and including yarn 3.0.0 dependencies is orthogonal. What matters more is the hdfs/common dependencies as they are shared between the timeline service unit tests and hbase. We could try to enforce 2.5.1 on mapreduce job client, but this has a potential of making the pom much bigger. Since all hadoop dependency versions are managed via dependency management (hadoop-project/pom.xml), we need to exclude it first, and declare a new dependency. But we may need to declare its dependencies too (as they are also managed, and without it 3.0.0 will be used, etc.). Mapreduce-client is near the top of the dependency chain, and the number of dependencies that needs to be replaced in this manner will be big. I haven't tried it, but that's my suspicion. My guess is that our unit tests are not exercising the mapreduce job client and that's probably why there are no issues. Let me know what you think. > hbase unit tests fail due to dependency issues > ---------------------------------------------- > > Key: YARN-5045 > URL: https://issues.apache.org/jira/browse/YARN-5045 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver > Affects Versions: YARN-2928 > Reporter: Sangjin Lee > Assignee: Sangjin Lee > Priority: Blocker > Attachments: YARN-5045-YARN-2928.01.patch, YARN-5045-YARN-2928.poc.patch > > > After the 5/4 rebase, the hbase unit tests in the timeline service project are failing: > {noformat} > org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage Time elapsed: 5.103 sec <<< ERROR! > java.io.IOException: Shutting down > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:423) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:356) > at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677) > at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546) > at org.apache.hadoop.hbase.http.HttpServer.(HttpServer.java:500) > at org.apache.hadoop.hbase.http.HttpServer.(HttpServer.java:104) > at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345) > at org.apache.hadoop.hbase.http.InfoServer.(InfoServer.java:77) > at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697) > at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:550) > at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:333) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:525) > at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139) > at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:217) > at org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:153) > at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:213) > at org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:93) > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:978) > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:938) > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812) > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806) > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:750) > at org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:87) > {noformat} > The root cause is that the hbase mini server depends on hadoop common's {{MetricsServlet}} which has been removed in the trunk (HADOOP-12504): > {noformat} > Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/metrics/MetricsServlet > at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677) > at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546) > at org.apache.hadoop.hbase.http.HttpServer.(HttpServer.java:500) > at org.apache.hadoop.hbase.http.HttpServer.(HttpServer.java:104) > at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345) > at org.apache.hadoop.hbase.http.InfoServer.(InfoServer.java:77) > at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697) > at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:550) > at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:333) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:525) > at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139) > ... 26 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org