hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dr. Christian Betz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7711) Error Serializing GenericUDF
Date Thu, 14 Aug 2014 09:32:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096789#comment-14096789
] 

Dr. Christian Betz commented on HIVE-7711:
------------------------------------------

ok, got my UDF to be usable with the config hive.plan.serialization.format=javaXML (after
digging through the sourcecode).

I did find the following snippet in https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java,
line 934:
    removeField(kryo, Operator.class, "colExprMap");
which is "missing"(?) in cloningQueryPlanKryo (lines 947-958).

IMHO, the best solution to this problem would involve adding appropriate serializers to Kryo,
e.g. from the https://github.com/magro/kryo-serializers project (not in my case, but just
as an example). To use this, one needs to expose the Kryo registry (not so good) or (in order
to be thread-safe) extend the registry with self-registered serializers. One way could expose
those serializers in GenericUDF::getSerializerMap to be overwritten, mapping classes to serializers.
However, this could result in major security issues I'm not yet aware of.

> Error Serializing GenericUDF
> ----------------------------
>
>                 Key: HIVE-7711
>                 URL: https://issues.apache.org/jira/browse/HIVE-7711
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.13.0
>            Reporter: Dr. Christian Betz
>
> I get an exception running a job with a GenericUDF in HIVE 0.13.0 (which was ok in HIVE
0.12.0).
> The org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc is serialized using Kryo,
trying to serialize stuff in my GenericUDF which is not serializable (doesn't implement Serializable).
> Switching to Kryo made the comment in ExprNodeGenericFuncDesc obsolte:
> "/**
>    * In case genericUDF is Serializable, we will serialize the object.
>    *
>    * In case genericUDF does not implement Serializable, Java will remember the
>    * class of genericUDF and creates a new instance when deserialized. This is
>    * exactly what we want.
>    */"
> Find the stacktrace below, however, the description above should be clear.
> Exception in thread "main" org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
> Serialization trace:
> value (java.util.concurrent.atomic.AtomicReference)
> state (clojure.lang.Atom)
> state (udfs.ArraySum)
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:918)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:826)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:840)
> 	at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:738)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.UnsupportedOperationException
> 	at clojure.lang.APersistentMap.put(APersistentMap.java:287)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:140)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	... 41 more



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message