hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10304) Running an hbase job jar: IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString
Date Fri, 10 Jan 2014 21:56:51 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868362#comment-13868362
] 

Nick Dimiduk commented on HBASE-10304:
--------------------------------------

Here's some copy we can use. Where in the book would you want something like this to live?
I also suggest the package-info be updated as well.

h3. Problem
Mapreduce jobs submitted to the cluster via a "fat jar," that is, a jar containing a 'lib'
directory with their runtime dependencies, fail to launch. The symptom is an exception similar
to the following:

{noformat}
Exception in thread "main" java.lang.IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString
cannot access its superclass com.google.protobuf.LiteralByteString
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:818)
	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.convertScanToString(TableMapReduceUtil.java:433)
	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:186)
	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:147)
	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:270)
	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:100)
...
{noformat}

This is because of an optimization introduced in [HBASE-9867|https://issues.apache.org/jira/browse/HBASE-9867]
that inadvertently introduced a classloader dependency.

Jobs submitted using a regular jar and specifying their runtime dependencies using the -libjars
parameter are not affected by this regression. More details about using the -libjars parameter
are available in this [blog post|http://grepalex.com/2013/02/25/hadoop-libjars/].

h3. Solution
In order to satisfy the new classloader requirements, hbase-protocol.jar must be included
in Hadoop's classpath. This can be resolved system-wide by including a reference to the hbase-protocol.jar
in hadoop's lib directory, via a symlink or by copying the jar into the new location.

This can also be achieved on a per-job launch basis by specifying a value for {{HADOOP_CLASSPATH}}
at job submission time. All three of the following job launching commands satisfy this requirement:

{noformat}
$ HADOOP_CLASSPATH=/path/to/hbase-protocol.jar hadoop jar MyJob.jar MyJobMainClass
$ HADOOP_CLASSPATH=$(hbase mapredcp) hadoop jar MyJob.jar MyJobMainClass
$ HADOOP_CLASSPATH=$(hbase classpath) hadoop jar MyJob.jar MyJobMainClass
{noformat}

h3. Apache Reference JIRA
See also [HBASE-10304|https://issues.apache.org/jira/browse/HBASE-10304].

> Running an hbase job jar: IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString
cannot access its superclass com.google.protobuf.LiteralByteString
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-10304
>                 URL: https://issues.apache.org/jira/browse/HBASE-10304
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce
>    Affects Versions: 0.98.0, 0.96.1.1
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.98.0
>
>         Attachments: hbase-10304_not_tested.patch, jobjar.xml
>
>
> (Jimmy has been working on this one internally.  I'm just the messenger raising this
critical issue upstream).
> So, if you make job jar and bundle up hbase inside in it because you want to access hbase
from your mapreduce task, the deploy of the job jar to the cluster fails with:
> {code}
> 14/01/05 08:59:19 INFO Configuration.deprecation: topology.node.switch.mapping.impl is
deprecated. Instead, use net.topology.node.switch.mapping.impl
> 14/01/05 08:59:19 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated.
Instead, use dfs.bytes-per-checksum
> Exception in thread "main" java.lang.IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString
cannot access its superclass com.google.protobuf.LiteralByteString
> 	at java.lang.ClassLoader.defineClass1(Native Method)
> 	at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
> 	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> 	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> 	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:818)
> 	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.convertScanToString(TableMapReduceUtil.java:433)
> 	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:186)
> 	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:147)
> 	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:270)
> 	at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:100)
> 	at com.ngdata.hbaseindexer.mr.HBaseMapReduceIndexerTool.run(HBaseMapReduceIndexerTool.java:124)
> 	at com.ngdata.hbaseindexer.mr.HBaseMapReduceIndexerTool.run(HBaseMapReduceIndexerTool.java:64)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at com.ngdata.hbaseindexer.mr.HBaseMapReduceIndexerTool.main(HBaseMapReduceIndexerTool.java:51)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> So, ZCLBS is a hack.  This class is in the hbase-protocol module.  It is "in" the com.google.protobuf
package.  All is well and good usually.
> But when we make a job jar and bundle up hbase inside it, our 'trick' breaks.  RunJar
makes a new class loader to run the job jar.  This URLCLassLoader 'attaches' all the jars
and classes that are in jobjar so they can be found when it does to do a lookup only Classloaders
work by always delegating to their parent first (unless you are a WAR file in a container
where delegation is 'off' for the most part) and in this case, the parent classloader will
have access to a pb jar since pb is in the hadoop CLASSPATH.  So, the parent loads the pb
classes.
> We then load ZCLBS only this is done in the claslsloader made by RunJar; ZKCLBS has a
different classloader from its superclass and we get the above IllegalAccessError.
> Now (Jimmy's work comes in here), this can't be fixed by reflection -- you can't setAccess
on a 'Class' -- and though it probably could be fixed by hacking RunJar so it was somehow
made configurable so we could put in place our own ClassLoader to do something like containers
do for WAR files (probably not a bad idea), there would be some fierce hackery involved and
besides, this won't show up in hadoop anytime too soon leaving hadoop 2.2ers out in the cold.
> So, the alternatives are:
> 1. Undo the ZCLSB hack.  We'd lose a lot of nice perf improvement but I'd say this is
preferable to crazy CLASSPATH hacks.
> 2. Require folks put hbase-protocol -- thats all you'd need -- on the hadoop CLASSPATH.
 This is kinda crazy.
> 3. We could try shading the pb jar content or probably better, just pull pb into hbase
altogether only under a different package.  If it was in our code base, we could do more ZCLSB-like
speedups.
> I was going to experiment with #3 above unless anyone else has a better idea.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message