hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
Date Tue, 14 Jun 2016 00:28:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328706#comment-15328706
] 

Sergey Shelukhin commented on HIVE-13930:
-----------------------------------------

Looks like all spark tests fail with this:
{noformat}
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- java.lang.NoClassDefFoundError: org/apache/hadoop/crypto/key/KeyProvider
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.lang.Class.forName0(Native Method)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.lang.Class.forName(Class.java:260)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:143)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:138)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:95)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.io.AcidUtils.<clinit>(AcidUtils.java:115)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.plan.TableScanDesc.<init>(TableScanDesc.java:129)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.plan.TableScanDesc.<init>(TableScanDesc.java:122)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.plan.TableScanDesc.<init>(TableScanDesc.java:113)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.plan.TableScanDescConstructorAccess.newInstance(Unknown Source)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy$1.newInstance(Kryo.java:1237)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1090)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:205)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.exec.spark.KryoSerializer.deserialize(KryoSerializer.java:60)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:317)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2016-06-11T15:51:36,750 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.lang.Thread.run(Thread.java:745)
2016-06-11T15:51:36,751 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.crypto.key.KeyProvider
2016-06-11T15:51:36,751 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
2016-06-11T15:51:36,751 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
2016-06-11T15:51:36,751 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.security.AccessController.doPrivileged(Native Method)
2016-06-11T15:51:36,751 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
2016-06-11T15:51:36,751 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
2016-06-11T15:51:36,751 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
2016-06-11T15:51:36,751 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2016-06-11T15:51:36,751 INFO  [stderr-redir-1[]]: client.SparkClientImpl (SparkClientImpl.java:run(616))
- 	... 44 more
{noformat}
It seems like dependency is there in q file project; the file has not moved since Hadoop 2.6
[~xuefuz] do you have any input on why this would fail? Or is this a known error, and the
cause of the failures is something else? The test seems to continue executing for a while
after this error.

I am looking at other tests. Most seem to be caused by YARN module restructuring.

Nm, this error is probably a red herring. Some dependency in ql is missing

> upgrade Hive to latest Hadoop version
> -------------------------------------
>
>                 Key: HIVE-13930
>                 URL: https://issues.apache.org/jira/browse/HIVE-13930
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-13930.01.patch, HIVE-13930.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message