hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6873) Clean up Coprocessor loading failure handling
Date Thu, 16 Jan 2014 21:19:22 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873957#comment-13873957
] 

Andrew Purtell commented on HBASE-6873:
---------------------------------------

I checked the build report of HBase-TRUNK #4825 (https://builds.apache.org/job/HBase-TRUNK/4825/)
. This is an occasional failure with TestHBaseFsck that turns up now and again and is not
related.

> Clean up Coprocessor loading failure handling
> ---------------------------------------------
>
>                 Key: HBASE-6873
>                 URL: https://issues.apache.org/jira/browse/HBASE-6873
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Coprocessors, regionserver
>    Affects Versions: 0.98.0
>            Reporter: David Arthur
>            Assignee: Andrew Purtell
>            Priority: Blocker
>             Fix For: 0.98.0, 0.99.0
>
>         Attachments: 6873.patch, 6873.patch, 6873.patch, 6873.patch, 6873.patch, 6873.patch,
6873.patch
>
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck
in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g., 
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is
not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.',
STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed,
marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting
to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully
transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting
to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully
transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.',
STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor
needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.,
starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> 	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> 	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> 	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> 	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> 	... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> 	at java.lang.Class.getDeclaredConstructors0(Native Method)
> 	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> 	at java.lang.Class.getConstructor0(Class.java:2699)
> 	at java.lang.Class.newInstance0(Class.java:326)
> 	at java.lang.Class.newInstance(Class.java:308)
> 	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> 	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> 	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> 	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> 	... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> 	... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.',
STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed,
marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting
to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully
transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting
to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully
transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.',
STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor
needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.,
starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> 	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> 	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> 	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> 	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> 	... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> 	at java.lang.Class.getDeclaredConstructors0(Native Method)
> 	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> 	at java.lang.Class.getConstructor0(Class.java:2699)
> 	at java.lang.Class.newInstance0(Class.java:326)
> 	at java.lang.Class.newInstance(Class.java:308)
> 	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> 	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> 	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> 	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> 	... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> 	... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor
should be unregister or at least disabled/blacklisted.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message