Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9496D200D70 for ; Fri, 22 Dec 2017 07:03:28 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 9340B160C2C; Fri, 22 Dec 2017 06:03:28 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B39D0160C2B for ; Fri, 22 Dec 2017 07:03:27 +0100 (CET) Received: (qmail 36108 invoked by uid 500); 22 Dec 2017 06:03:21 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Delivered-To: moderator for user@hbase.apache.org Received: (qmail 91351 invoked by uid 99); 22 Dec 2017 01:36:59 -0000 X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -5.111 X-Spam-Level: X-Spam-Status: No, score=-5.111 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=llnl.gov DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=llnl.gov; i=@llnl.gov; q=dns/txt; s=08082017; t=1513906617; x=1545442617; h=message-id:subject:from:to:date:mime-version: content-transfer-encoding; bh=IUxdfQnCW9J47CKXBIJg5PqG3XiyRiiOYniD0AUog40=; b=jTzMWs81lF3pTrH2GdOHAQE1bIbW/dYTiEEhHG6ttLJvYN61+cLH+vZT RfAkQTv8EKVwinNzR7V2pwUQwt3+eXLbX0xZyjr8nUa0lQF76snzNJK3L i3xOukvtzN3MGcct30F4BNLQSEtjNkv07RwJJx6jQRAO0aP4jdiqxWYXl I=; X-Attachments: Message-ID: <1513906609.3389.27.camel@llnl.gov> Subject: Error running PerformanceEvaluation in Hbase 1.4.0 (works in prior versions) From: Albert Chu To: user@hbase.apache.org Date: Thu, 21 Dec 2017 17:36:49 -0800 Organization: LLNL Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.22.6 (3.22.6-10.el7) Mime-Version: 1.0 Content-Transfer-Encoding: 8bit archived-at: Fri, 22 Dec 2017 06:03:28 -0000 Hi, I have a unit test that runs a simple PerformanceEvaluation test to make sure things are basically working bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=50000 sequentialWrite 1 This test runs against Hadoop 2.7.0 and works against Hbase 0.99.0 through 1.3.1. But it broke with 1.4.0 with the error below. The errors are coming from within the map tasks.  Suggesting that they  are missing some jar/class dependency.  Important dependencies were automatically handled before, but an issue now arises. I've tried adding various jars to HBASE_CLASSPATH to no avail. So some subtlety is being missed. 2017-12-21 13:49:40,974 INFO  [main] mapreduce.Job: Task Id : attempt_1513892752187_0002_m_000004_2, Status : FAILED Error: java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119) at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:297) at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:250) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) ... 12 more Caused by: java.lang.RuntimeException: Could not create  interface org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop compatibility jar on the classpath? at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75) at org.apache.hadoop.hbase.zookeeper.MetricsZooKeeper.(MetricsZooKeeper.java:38) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:130) at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:143) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:181) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:155) at org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.(ZooKeeperKeepAliveConnection.java:43) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1737) at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:104) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:945) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:721) ... 17 more Caused by: java.util.ServiceConfigurationError: org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:224) at java.util.ServiceLoader.access$100(ServiceLoader.java:181) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377) at java.util.ServiceLoader$1.next(ServiceLoader.java:445) at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59) ... 27 more Caused by: java.lang.NoClassDefFoundError: Lorg/apache/hadoop/hbase/metrics/MetricRegistry; at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2509) at java.lang.Class.getDeclaredFields(Class.java:1819) at org.apache.hadoop.util.ReflectionUtils.getDeclaredFieldsIncludingInherited(ReflectionUtils.java:323) at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder.initRegistry(MetricsSourceBuilder.java:92) at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder.(MetricsSourceBuilder.java:56) at org.apache.hadoop.metrics2.lib.MetricsAnnotations.newSourceBuilder(MetricsAnnotations.java:42) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:224) at org.apache.hadoop.hbase.metrics.BaseSourceImpl.(BaseSourceImpl.java:115) at org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.(MetricsZooKeeperSourceImpl.java:56) at org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.(MetricsZooKeeperSourceImpl.java:51) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:383) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373) ... 29 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.metrics.MetricRegistry at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 46 more -- Albert Chu chu11@llnl.gov Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory