hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
Date Thu, 11 Jan 2018 17:47:00 GMT
Josh Elser created HBASE-19769:
----------------------------------

             Summary: IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
jobs
                 Key: HBASE-19769
                 URL: https://issues.apache.org/jira/browse/HBASE-19769
             Project: HBase
          Issue Type: Bug
          Components: mapreduce, metrics
    Affects Versions: 2.0.0-beta-1
            Reporter: Josh Elser
            Assignee: Josh Elser
            Priority: Critical
             Fix For: 2.0.0-beta-2


issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893

Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has been using a custom
classloader to separate Hadoop dependencies from the user's JAR being run. A separate classloader
is created for the user-provided jar, and then this classloader is set as the contextClassLoader
before the Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for us
to date because we don't try to access any Hadoop internal classes client-side.

However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The problem is that
Hadoop metrics2 implementations which we reference from the same package are loaded by a different
classloader than our HBase code is loaded from. This makes the expected package-private access
of these Metrics2 classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.

{noformat}
java.lang.RuntimeException: Could not create  interface org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource
Is the hadoop compatibility jar on the classpath?
        at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
        at org.apache.hadoop.hbase.zookeeper.ZKMetrics.<init>(ZKMetrics.java:36)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:115)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:128)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:102)
        at org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
        at org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
        at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
        at org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
        at org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
        at org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
        at org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.util.ServiceConfigurationError: org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource:
Provider org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:232)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
        at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
        ... 21 more
Caused by: java.lang.IllegalAccessError: tried to access class org.apache.hadoop.metrics2.lib.MetricsInfoImpl
from class org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry
        at org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry.newGauge(DynamicMetricsRegistry.java:139)
        at org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:59)
        at org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:51)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
        ... 24 more
{noformat}

There are two ways to fix this problem that I see now:

# Remove client-side ZK metrics and disallow any more client-side metrics that directly use
metrics2 API
# Use public API for metrics2 only in HBase, or copy implementations from Hadoop to avoid
the need to access the package-private classes from this goofy state.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message