Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 58BB417E40 for ; Tue, 4 Nov 2014 01:10:36 +0000 (UTC) Received: (qmail 306 invoked by uid 500); 4 Nov 2014 01:10:36 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 257 invoked by uid 500); 4 Nov 2014 01:10:36 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 242 invoked by uid 99); 4 Nov 2014 01:10:36 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2014 01:10:36 +0000 Date: Tue, 4 Nov 2014 01:10:36 +0000 (UTC) From: "Jing Zhou (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-9936) Table get stuck when it fails to open due to a coprocessor error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-9936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195511#comment-14195511 ] Jing Zhou commented on HBASE-9936: ---------------------------------- I also met this problem. And I found that my endpoint jar dependencies conflict with my custom filter comparator jar dependencies. So I built endpoint jar without dependencies, and it worked. > Table get stuck when it fails to open due to a coprocessor error > ---------------------------------------------------------------- > > Key: HBASE-9936 > URL: https://issues.apache.org/jira/browse/HBASE-9936 > Project: HBase > Issue Type: Bug > Affects Versions: 0.96.0 > Reporter: Benoit Sigoure > > I made a mistake while after re-enabling a table on which I did an `alter' to add a coprocessor: the .jar I specified wasn't a self-contained jar, and thus some dependent classes couldn't be found. > {code} > 2013-11-09 02:39:05,994 INFO [AM.ZK.Worker-pool2-t17] master.RegionStates: Transitioned {8568640c1da6ce0d5e27b656d28fe9fd state=PENDING_OPEN, ts=1383993545988, server=192.168.42.108,59570,1383993435386} to {8568640c1da6ce0d5e27b656d28fe9fd state=OPENING, ts=1383993545994, server=192.168.42.108,59570,1383993435386} > 2013-11-09 02:39:05,995 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] coprocessor.CoprocessorHost: Loading coprocessor class com.example.foo.hbase.FooCoprocessor with path /Users/tsuna/src/foo/target/scala-2.10/foo_2.10-0.1.jar and priority 1000 > 2013-11-09 02:39:06,005 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader: Finding class: com.example.foo.hbase.FooCoprocessor > 2013-11-09 02:39:06,006 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader: Skipping exempt class org.apache.hadoop.hbase.coprocessor.BaseRegionObserver - delegating directly to parent > 2013-11-09 02:39:06,007 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader: Skipping exempt class java.lang.Object - delegating directly to parent > 2013-11-09 02:39:06,007 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader: Finding class: org.slf4j.LoggerFactory > 2013-11-09 02:39:06,007 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader: Class org.slf4j.LoggerFactory not found - delegating to parent > 2013-11-09 02:39:06,008 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader: Finding class: scala.collection.mutable.StringBuilder > 2013-11-09 02:39:06,008 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader: Class scala.collection.mutable.StringBuilder not found - delegating to parent > 2013-11-09 02:39:06,008 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader: Class scala.collection.mutable.StringBuilder not found in parent loader > 2013-11-09 02:39:06,008 ERROR [RS_OPEN_REGION-192.168.42.108:59570-2] handler.OpenRegionHandler: Failed open of region=foo,,1383899959121.8568640c1da6ce0d5e27b656d28fe9fd., starting to roll back the global memstore size. > java.lang.IllegalStateException: Could not instantiate a region instance. > at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3820) > at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4078) > at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4030) > at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3981) > at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:475) > at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:140) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3817) > ... 9 more > Caused by: java.lang.NoClassDefFoundError: scala/collection/mutable/StringBuilder > at com.example.foo.hbase.FooCoprocessor.start(FooCoprocessor.scala:18) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:636) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:259) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:212) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:192) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:154) > at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:532) > at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:442) > ... 14 more > Caused by: java.lang.ClassNotFoundException: scala.collection.mutable.StringBuilder > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:299) > ... 22 more > 2013-11-09 02:39:06,010 INFO [RS_OPEN_REGION-192.168.42.108:59570-2] handler.OpenRegionHandler: Opening of region {ENCODED => 8568640c1da6ce0d5e27b656d28fe9fd, NAME => 'foo,,1383899959121.8568640c1da6ce0d5e27b656d28fe9fd.', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 1 > {code} > The master retried 10 times and then seems to have given up. I'm trying to point it to a different jar, but I can't, and the table now seems stuck in a not-enabled-nor-disabled state: > {code} > hbase(main):002:0> disable 'foo' > ERROR: org.apache.hadoop.hbase.TableNotEnabledException: foo > at org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100) > at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979) > at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990) > at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38217) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146) > at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851) > Here is some help for this command: > Start disable of named table: e.g. "hbase> disable 't1'" > hbase(main):003:0> enable 'foo' > ERROR: org.apache.hadoop.hbase.TableNotDisabledException: foo > at org.apache.hadoop.hbase.master.handler.EnableTableHandler.prepare(EnableTableHandler.java:109) > at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:1954) > at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:1965) > at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38215) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146) > at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851) > Here is some help for this command: > Start enable of named table: e.g. "hbase> enable 't1'" > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)