hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4946) HTable.coprocessorExec (and possibly coprocessorProxy) does not work with dynamically loaded coprocessors (from hdfs or local system), because the RPC system tries to deserialize an unknown class.
Date Tue, 06 Dec 2011 21:07:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163840#comment-13163840

Ted Yu commented on HBASE-4946:

For coprocessor/Exec.java, javadoc doesn't match code:
     try {
       protocol = (Class<CoprocessorProtocol>)conf.getClassByName(protocolName);
     catch (ClassNotFoundException cnfe) {
-      throw new IOException("Protocol class "+protocolName+" not found", cnfe);
+      // can't do eager instantiation. pass it as a string and try to deserialize later.
+      //throw new IOException("Protocol class "+protocolName+" not found", cnfe);
I think the above try block should be commented out.
TestCoprocessorEndpoint passes without the above assignment to protocol.
> HTable.coprocessorExec (and possibly coprocessorProxy) does not work with dynamically
loaded coprocessors (from hdfs or local system), because the RPC system tries to deserialize
an unknown class. 
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>                 Key: HBASE-4946
>                 URL: https://issues.apache.org/jira/browse/HBASE-4946
>             Project: HBase
>          Issue Type: Bug
>          Components: coprocessors
>    Affects Versions: 0.92.0
>            Reporter: Andrei Dragomir
>            Assignee: Andrei Dragomir
>         Attachments: HBASE-4946-v2.patch, HBASE-4946-v3.patch, HBASE-4946.patch
> Loading coprocessors jars from hdfs works fine. I load it from the shell, after setting
the attribute, and it gets loaded:
> {noformat}
> INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config
now ...
> INFO org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Class com.MyCoprocessorClass
needs to be loaded from a file - hdfs://localhost:9000/coproc/rt-      >0.0.1-SNAPSHOT.jar.
> INFO org.apache.hadoop.hbase.coprocessor.CoprocessorHost: loadInstance: com.MyCoprocessorClass
> INFO org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: RegionEnvironment createEnvironment
> DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol handler: region=t1,,1322572939753.6409aee1726d31f5e5671a59fe6e384f.
> INFO org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Load coprocessor com.MyCoprocessorClass
from HTD of t1 successfully.
> {noformat}
> The problem is that this coprocessors simply extends BaseEndpointCoprocessor, with a
dynamic method. When calling this method from the client with HTable.coprocessorExec, I get
errors on the HRegionServer, because the call cannot be deserialized from writables. 
> The problem is that Exec tries to do an "early" resolve of the coprocessor class. The
coprocessor class is loaded, but it is in the context of the HRegionServer / HRegion. So,
the call fails:
> {noformat}
> 2011-12-02 00:34:17,348 ERROR org.apache.hadoop.hbase.io.HbaseObjectWritable: Error in
> java.io.IOException: Protocol class com.MyCoprocessorClassProtocol not found
>   at org.apache.hadoop.hbase.client.coprocessor.Exec.readFields(Exec.java:125)
>   at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:575)
>   at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:105)
>   at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1237)
>   at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1167)
>   at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:703)
>   at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:495)
>   at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>   at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.ClassNotFoundException: com.MyCoprocessorClassProtocol
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:247)
>   at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
>   at org.apache.hadoop.hbase.client.coprocessor.Exec.readFields(Exec.java:122)
>   ... 10 more
> {noformat}
> Probably the correct way to fix this is to make Exec really smart, so that it knows all
the class definitions loaded in CoprocessorHost(s).
> I created a small patch that simply doesn't resolve the class definition in the Exec,
instead passing it as string down to the HRegion layer. This layer knows all the definitions,
and simply loads it by name. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message