hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navis류승우 <navis....@nexr.com>
Subject Re: UDF related: org.apache.hive.com.esotericsoftware.kryo.KryoException
Date Tue, 21 Oct 2014 07:00:50 GMT
The states in UDF(which is represented as "states SomeClass3") are needed?
If it's not, you can try them mark as a transient field.

Thanks,
Navis

2014-10-20 23:59 GMT+09:00 Harel Gliksman <harelglik@gmail.com>:

> Hello,
>
> I am experiencing inconsistent behavior when trying to use UDF on 0.13.1
> on Amazon's EMR (AMI 3.2.1).
>
> I generated a uber jar and deployed a UDF like so:
>
> create temporary function someFunction as "hive.udf.localization.MyUDF"
> using jar "s3://waze.mapreduce.shared/scripts/Hive/MyHive.jar";
>
> I am having these 2 (related?) problems:
>
> 1) When I simply try to use my UDF I get
>
> Error: java.lang.RuntimeException:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NullPointerException
> Serialization trace:
> childRectangles (SomeClass1)
> statesTree (SomeClass2)
> states (SomeClass3)
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:360)
> at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:271)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:431)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:410)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NullPointerException...
>
> After digging I found this:
>
>
> http://mail-archives.apache.org/mod_mbox/hive-dev/201408.mbox/%3CJIRA.12733732.1407927053435.81293.1408008733570@arcas%3E
>
> *and after setting *
>
> *hive.plan.serialization.format=javaXML*
>
> *the UDF is running OK on my test data set of 50 lines.*
>
> 2) When running the UDF in a more complex 2-joins query I am getting a
> somewhat related error:
>
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NullPointerException
> Serialization trace:
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: java.lang.NullPointerException
> Serialization trace:
> childRectangles (SomeClass1)
> statesTree (SomeClass2)
> states (SomeClass3)
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator)
> opParseCtxMap (org.apache.hadoop.hive.ql.plan.MapWork)
> mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)...
>
> *This does not go away even after
> setting hive.plan.serialization.format=javaXML *
>
> Can someone please advise?
> Many thanks,
> Harel.
>

Mime
View raw message