hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-9936) Table get stuck when it fails to open due to a coprocessor error
Date Tue, 04 Nov 2014 01:10:36 GMT

    [ https://issues.apache.org/jira/browse/HBASE-9936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195511#comment-14195511
] 

Jing Zhou commented on HBASE-9936:
----------------------------------

I also met this problem. And I found that my endpoint jar dependencies conflict with my custom
filter comparator jar dependencies. So I built endpoint jar without dependencies, and it worked.


> Table get stuck when it fails to open due to a coprocessor error
> ----------------------------------------------------------------
>
>                 Key: HBASE-9936
>                 URL: https://issues.apache.org/jira/browse/HBASE-9936
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.96.0
>            Reporter: Benoit Sigoure
>
> I made a mistake while after re-enabling a table on which I did an `alter' to add a coprocessor:
the .jar I specified wasn't a self-contained jar, and thus some dependent classes couldn't
be found.
> {code}
> 2013-11-09 02:39:05,994 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transitioned
{8568640c1da6ce0d5e27b656d28fe9fd state=PENDING_OPEN, ts=1383993545988, server=192.168.42.108,59570,1383993435386}
to {8568640c1da6ce0d5e27b656d28fe9fd state=OPENING, ts=1383993545994, server=192.168.42.108,59570,1383993435386}
> 2013-11-09 02:39:05,995 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] coprocessor.CoprocessorHost:
Loading coprocessor class com.example.foo.hbase.FooCoprocessor with path /Users/tsuna/src/foo/target/scala-2.10/foo_2.10-0.1.jar
and priority 1000
> 2013-11-09 02:39:06,005 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader:
Finding class: com.example.foo.hbase.FooCoprocessor
> 2013-11-09 02:39:06,006 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader:
Skipping exempt class org.apache.hadoop.hbase.coprocessor.BaseRegionObserver - delegating
directly to parent
> 2013-11-09 02:39:06,007 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader:
Skipping exempt class java.lang.Object - delegating directly to parent
> 2013-11-09 02:39:06,007 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader:
Finding class: org.slf4j.LoggerFactory
> 2013-11-09 02:39:06,007 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader:
Class org.slf4j.LoggerFactory not found - delegating to parent
> 2013-11-09 02:39:06,008 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader:
Finding class: scala.collection.mutable.StringBuilder
> 2013-11-09 02:39:06,008 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader:
Class scala.collection.mutable.StringBuilder not found - delegating to parent
> 2013-11-09 02:39:06,008 DEBUG [RS_OPEN_REGION-192.168.42.108:59570-2] util.CoprocessorClassLoader:
Class scala.collection.mutable.StringBuilder not found in parent loader
> 2013-11-09 02:39:06,008 ERROR [RS_OPEN_REGION-192.168.42.108:59570-2] handler.OpenRegionHandler:
Failed open of region=foo,,1383899959121.8568640c1da6ce0d5e27b656d28fe9fd., starting to roll
back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
>         at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3820)
>         at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4078)
>         at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4030)
>         at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3981)
>         at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:475)
>         at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:140)
>         at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>         at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3817)
>         ... 9 more
> Caused by: java.lang.NoClassDefFoundError: scala/collection/mutable/StringBuilder
>         at com.example.foo.hbase.FooCoprocessor.start(FooCoprocessor.scala:18)
>         at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:636)
>         at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:259)
>         at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:212)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:192)
>         at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:154)
>         at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:532)
>         at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:442)
>         ... 14 more
> Caused by: java.lang.ClassNotFoundException: scala.collection.mutable.StringBuilder
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>         at org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:299)
>         ... 22 more
> 2013-11-09 02:39:06,010 INFO  [RS_OPEN_REGION-192.168.42.108:59570-2] handler.OpenRegionHandler:
Opening of region {ENCODED => 8568640c1da6ce0d5e27b656d28fe9fd, NAME => 'foo,,1383899959121.8568640c1da6ce0d5e27b656d28fe9fd.',
STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK,
expecting version 1
> {code}
> The master retried 10 times and then seems to have given up.  I'm trying to point it
to a different jar, but I can't, and the table now seems stuck in a not-enabled-nor-disabled
state:
> {code}
> hbase(main):002:0> disable 'foo'
> ERROR: org.apache.hadoop.hbase.TableNotEnabledException: foo
> 	at org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 	at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 	at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> 	at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38217)
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> 	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
> Here is some help for this command:
> Start disable of named table: e.g. "hbase> disable 't1'"
> hbase(main):003:0> enable 'foo'
> ERROR: org.apache.hadoop.hbase.TableNotDisabledException: foo
> 	at org.apache.hadoop.hbase.master.handler.EnableTableHandler.prepare(EnableTableHandler.java:109)
> 	at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:1954)
> 	at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:1965)
> 	at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38215)
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
> 	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
> Here is some help for this command:
> Start enable of named table: e.g. "hbase> enable 't1'"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message