Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A8D12175AF for ; Mon, 20 Oct 2014 14:59:35 +0000 (UTC) Received: (qmail 20478 invoked by uid 500); 20 Oct 2014 14:59:33 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 20407 invoked by uid 500); 20 Oct 2014 14:59:33 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 20397 invoked by uid 99); 20 Oct 2014 14:59:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Oct 2014 14:59:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of harelglik@gmail.com designates 209.85.223.172 as permitted sender) Received: from [209.85.223.172] (HELO mail-ie0-f172.google.com) (209.85.223.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Oct 2014 14:59:28 +0000 Received: by mail-ie0-f172.google.com with SMTP id rl12so4858925iec.31 for ; Mon, 20 Oct 2014 07:59:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=aT5ns+IDxMlsnjJjuvR8XCY3rfPlHQp9ih40PYmW67g=; b=0Bwh+0OBMt6irwujhwQhG6UzKvmZYhPvNJDBcxOon58m/L4Q0T2GE3hDyb1Z9bouy+ nr/APW9apYfc0AXfR+fg8w9v3pYxCv1Git6yWrnkqSgqZH7D0yYFZ+4hMQ749/aSaP25 mmzImMGazsQA+zbFRW55STdIm89T49MH8E2vd7onJUkHC0MzRS4Km9P/LCHoK9H9Yk6k WEoRjR1uwEjkNb7dHDbImLIzu13g55uv/6A+GpVGrmVsPu1LVZKceRrm4jyynph3d5uF nTUif+28WdpGsYxKyWD+7GTRpIDnLaBwW9XUEIosTtmFFM+bAMn9lfClQoJs+7RPect2 89yA== MIME-Version: 1.0 X-Received: by 10.107.30.136 with SMTP id e130mr29873343ioe.9.1413817147635; Mon, 20 Oct 2014 07:59:07 -0700 (PDT) Received: by 10.107.160.195 with HTTP; Mon, 20 Oct 2014 07:59:07 -0700 (PDT) Date: Mon, 20 Oct 2014 17:59:07 +0300 Message-ID: Subject: UDF related: org.apache.hive.com.esotericsoftware.kryo.KryoException From: Harel Gliksman To: user@hive.apache.org Content-Type: multipart/alternative; boundary=001a1140eb20dcd5750505dbf4a5 X-Virus-Checked: Checked by ClamAV on apache.org --001a1140eb20dcd5750505dbf4a5 Content-Type: text/plain; charset=UTF-8 Hello, I am experiencing inconsistent behavior when trying to use UDF on 0.13.1 on Amazon's EMR (AMI 3.2.1). I generated a uber jar and deployed a UDF like so: create temporary function someFunction as "hive.udf.localization.MyUDF" using jar "s3://waze.mapreduce.shared/scripts/Hive/MyHive.jar"; I am having these 2 (related?) problems: 1) When I simply try to use my UDF I get Error: java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: childRectangles (SomeClass1) statesTree (SomeClass2) states (SomeClass3) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:360) at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:271) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:431) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:169) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:410) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException... After digging I found this: http://mail-archives.apache.org/mod_mbox/hive-dev/201408.mbox/%3CJIRA.12733732.1407927053435.81293.1408008733570@arcas%3E *and after setting * *hive.plan.serialization.format=javaXML* *the UDF is running OK on my test data set of 50 lines.* 2) When running the UDF in a more complex 2-joins query I am getting a somewhat related error: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task Error: java.lang.NullPointerException Serialization trace: childRectangles (SomeClass1) statesTree (SomeClass2) states (SomeClass3) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) opParseCtxMap (org.apache.hadoop.hive.ql.plan.MapWork) mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)... *This does not go away even after setting hive.plan.serialization.format=javaXML * Can someone please advise? Many thanks, Harel. --001a1140eb20dcd5750505dbf4a5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello,

I am experiencing inconsistent b= ehavior when trying to use UDF on 0.13.1 on Amazon's EMR (AMI=C2=A03.2.1).

I generated a uber jar an= d deployed a UDF like so:

create temporary functio= n someFunction as "hive.udf.localization.MyUDF" using jar "s= 3://waze.mapreduce.shared/scripts/Hive/MyHive.jar";

I am having these 2 (related?) problems:

1)= When I simply try to use my UDF I get=C2=A0

Error: java= .lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoExcep= tion: java.lang.NullPointerException
Serialization trace:
childRectan= gles (SomeClass1)
statesTree (SomeClass2)
states (SomeClass3)
gene= ricUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprM= ap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.a= pache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.had= oop.hive.ql.plan.MapWork)
at org.apache.hadoop.hive.ql.exec.Utilities.ge= tBaseWork(Utilities.java:360)
at org.apache.hadoop.hive.ql.exec.Utilitie= s.getMapWork(Utilities.java:271)
at org.apache.hadoop.hive.ql.io.HiveInp= utFormat.init(HiveInputFormat.java:254)
at org.apache.hadoop.hive.ql.io.= HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
at o= rg.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveI= nputFormat.java:431)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFor= mat.getRecordReader(CombineHiveInputFormat.java:587)
at org.apache.hadoo= p.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at o= rg.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:410)
at org.ap= ache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.ma= pred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessControll= er.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subje= ct.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(Use= rGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(= YarnChild.java:162)
Caused by: org.apache.hive.com.esotericsoftware.kryo= .KryoException: java.lang.NullPointerException...

After diggi= ng I found this:

http://mail-archives.apache.org/mod_mbox/hive-dev/201408.= mbox/%3CJIRA.12733732.1407927053435.81293.1408008733570@arcas%3E

and after setting=C2=A0

hive.plan.serialization.format=3DjavaXML

th= e UDF is running OK on my test data set of 50 lines.

2) When running the UDF in a more complex 2-joins query I am getting a s= omewhat related error:

org.apache.hive.com.esotericsoftware.kryo.KryoException: jav= a.lang.NullPointerException
Serialization trace:
org.apache.hadoop.hi= ve.ql.parse.SemanticException: Generate Map Join Task Error: java.lang.Null= PointerException
Serialization trace:
childRectangles (SomeClass1)
statesTree (SomeClass2)
<= span style=3D"background-color:rgb(244,204,204)">states (SomeClass3)=
genericUDF (org.apach= e.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.ha= doop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive= .ql.exec.JoinOperator)
opParseCtxMap (org.apache.hadoop.hive.ql.plan.Map= Work)
mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)...
<= /p>

This does not go away even after setting=C2=A0hive.p= lan.serialization.format=3DjavaXML=C2=A0

Can someo= ne please advise?
Many thanks,
Harel.

--001a1140eb20dcd5750505dbf4a5--