hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harel Gliksman <harelg...@gmail.com>
Subject UDF related: org.apache.hive.com.esotericsoftware.kryo.KryoException
Date Mon, 20 Oct 2014 14:59:07 GMT
Hello,

I am experiencing inconsistent behavior when trying to use UDF on 0.13.1 on
Amazon's EMR (AMI 3.2.1).

I generated a uber jar and deployed a UDF like so:

create temporary function someFunction as "hive.udf.localization.MyUDF"
using jar "s3://waze.mapreduce.shared/scripts/Hive/MyHive.jar";

I am having these 2 (related?) problems:

1) When I simply try to use my UDF I get

Error: java.lang.RuntimeException:
org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.NullPointerException
Serialization trace:
childRectangles (SomeClass1)
statesTree (SomeClass2)
states (SomeClass3)
genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:360)
at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:271)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:431)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:410)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.NullPointerException...

After digging I found this:

http://mail-archives.apache.org/mod_mbox/hive-dev/201408.mbox/%3CJIRA.12733732.1407927053435.81293.1408008733570@arcas%3E

*and after setting *

*hive.plan.serialization.format=javaXML*

*the UDF is running OK on my test data set of 50 lines.*

2) When running the UDF in a more complex 2-joins query I am getting a
somewhat related error:

org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.NullPointerException
Serialization trace:
org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
Error: java.lang.NullPointerException
Serialization trace:
childRectangles (SomeClass1)
statesTree (SomeClass2)
states (SomeClass3)
genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator)
opParseCtxMap (org.apache.hadoop.hive.ql.plan.MapWork)
mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)...

*This does not go away even after
setting hive.plan.serialization.format=javaXML *

Can someone please advise?
Many thanks,
Harel.

Mime
View raw message