hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Dere (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11762) TestHCatLoaderEncryption failures when using Hadoop 2.7
Date Thu, 10 Sep 2015 21:38:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739647#comment-14739647
] 

Jason Dere commented on HIVE-11762:
-----------------------------------

Whoa, lot of failures .. I ran TestSparkCliDriver and see the following error:
{noformat}
2015-09-10 14:14:31,970 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
- Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/crypto/key/KeyProvider
2015-09-10 14:14:31,970 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.hadoop.hive.shims.Hadoop23Shims.<clinit>(Hadoop23Shims.java:1058)
2015-09-10 14:14:31,970 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at java.lang.Class.forName0(Native Method)
2015-09-10 14:14:31,970 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at java.lang.Class.forName(Class.java:190)
2015-09-10 14:14:31,970 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:146)
2015-09-10 14:14:31,970 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:141)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:369)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:46)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:146)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at java.lang.reflect.Method.invoke(Method.java:606)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
- Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.crypto.key.KeyProvider
2015-09-10 14:14:31,971 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
2015-09-10 14:14:31,972 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
2015-09-10 14:14:31,972 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at java.security.AccessController.doPrivileged(Native Method)
2015-09-10 14:14:31,972 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
2015-09-10 14:14:31,972 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
2015-09-10 14:14:31,972 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
2015-09-10 14:14:31,972 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
2015-09-10 14:14:31,972 INFO  [stderr-redir-1] client.SparkClientImpl (SparkClientImpl.java:run(588))
-         ... 19 more
{noformat}

> TestHCatLoaderEncryption failures when using Hadoop 2.7
> -------------------------------------------------------
>
>                 Key: HIVE-11762
>                 URL: https://issues.apache.org/jira/browse/HIVE-11762
>             Project: Hive
>          Issue Type: Bug
>          Components: Shims, Tests
>            Reporter: Jason Dere
>            Assignee: Jason Dere
>         Attachments: HIVE-11762.1.patch, HIVE-11762.2.patch
>
>
> When running TestHCatLoaderEncryption with -Dhadoop23.version=2.7.0, we get the following
error during setup():
> {noformat}
> testReadDataFromEncryptedHiveTableByPig[5](org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption)
 Time elapsed: 3.648 sec  <<< ERROR!
> java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.DFSClient.setKeyProvider(Lorg/apache/hadoop/crypto/key/KeyProviderCryptoExtension;)V
> 	at org.apache.hadoop.hive.shims.Hadoop23Shims.getMiniDfs(Hadoop23Shims.java:534)
> 	at org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.initEncryptionShim(TestHCatLoaderEncryption.java:252)
> 	at org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.setup(TestHCatLoaderEncryption.java:200)
> {noformat}
> It looks like between Hadoop 2.6 and Hadoop 2.7, the argument to DFSClient.setKeyProvider()
changed:
> {noformat}
>    @VisibleForTesting
> -  public void setKeyProvider(KeyProviderCryptoExtension provider) {
> -    this.provider = provider;
> +  public void setKeyProvider(KeyProvider provider) {
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message