hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengxiang Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]
Date Wed, 04 Feb 2015 00:34:35 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304382#comment-14304382
] 

Chengxiang Li commented on HIVE-9410:
-------------------------------------

Not actually, as you can see from the patch, i stored added jar paths to a list in JobContextImpl,
and add the jar paths in JobContextImpl to current thead context class loader while execute
JobStatusJob each time, as JobContextImpl is a singleton instance for RemoteDriver service,
so later request thead could get the jar paths as well. 

> ClassNotFoundException occurs during hive query case execution with UDF defined [Spark
Branch]
> ----------------------------------------------------------------------------------------------
>
>                 Key: HIVE-9410
>                 URL: https://issues.apache.org/jira/browse/HIVE-9410
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>         Environment: CentOS 6.5
> JDK1.7
>            Reporter: Xin Hao
>            Assignee: Chengxiang Li
>             Fix For: spark-branch, 1.1.0
>
>         Attachments: HIVE-9410.1-spark.patch, HIVE-9410.2-spark.patch, HIVE-9410.3-spark.patch,
HIVE-9410.4-spark.patch, HIVE-9410.4-spark.patch
>
>
> We have a hive query case with UDF defined (i.e. BigBench case Q10, Q18 etc.). It will
be passed for default Hive (on MR) mode, while failed for Hive On Spark mode (both Standalone
and Yarn-Client). 
> Although we use 'add jar XXXX.jar;' to add the UDF jar explicitly, the issue still exists.

> BTW, if we put the UDF jar into $HIVE_HOME/lib dir, the case will be passed.
> Detail Error Message is as below (NOTE: de.bankmark.bigbench.queries.q10.SentimentUDF
is the UDF which contained in jar bigbenchqueriesmr.jar, and we have add command like 'add
jar /location/to/bigbenchqueriesmr.jar;' into .sql explicitly)
> {code}
> INFO  [pool-1-thread-1]: client.RemoteDriver (RemoteDriver.java:call(316)) - Failed to
run job 8dd120cb-1a4d-4d1c-ba31-61eac648c27d
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: de.bankmark.bigbench.queries.q10.SentimentUDF
> Serialization trace:
> genericUDTF (org.apache.hadoop.hive.ql.plan.UDTFDesc)
> conf (org.apache.hadoop.hive.ql.exec.UDTFOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.MapJoinOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> right (org.apache.commons.lang3.tuple.ImmutablePair)
> edgeProperties (org.apache.hadoop.hive.ql.plan.SparkWork)
>         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
>         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
>         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
>         at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
>         at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
>         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
>         at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>         at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
>         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
>         at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
>         at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
>         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
>         at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>         at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
>         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
>         at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> ...
> Caused by: java.lang.ClassNotFoundException: de.bankmark.bigbench.queries.q10.SentimentUDF
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:270)
>         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
>         ... 55 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message