Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7B1BFC30D for ; Wed, 30 May 2012 16:14:39 +0000 (UTC) Received: (qmail 52167 invoked by uid 500); 30 May 2012 16:14:38 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 52120 invoked by uid 500); 30 May 2012 16:14:38 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 52111 invoked by uid 99); 30 May 2012 16:14:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 May 2012 16:14:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of praveenesh@gmail.com designates 209.85.212.182 as permitted sender) Received: from [209.85.212.182] (HELO mail-wi0-f182.google.com) (209.85.212.182) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 May 2012 16:14:31 +0000 Received: by wibhm6 with SMTP id hm6so2968125wib.5 for ; Wed, 30 May 2012 09:14:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=19ZpY8YGoU/GN/Y+StSXfjDwMldCUGtPU0Zx68TqdUw=; b=dd5mvsf117qD+Cl+iDtWZmpIVstREdB/4kXsxRW3RtzgAhMJSYPdxMqkcLtIEXGYME wBZaPl+fd9IFamsE5fHqK8BN06fZznI2D2/xa8CGweW32aIovCNC5SfMMGCoN5dNJDJV Fs3Tbkkw5lfezjjRLsMq7yBTrEEzdNQxbMJcv8v7i4uuG+FJ/TvdRVOw9JJEcgI4D+Nb Cmk0svn7e91AbBe/zZCkvLx0TiTH2B5b7Xih3A0P3Bj/ETtald4gBrcCMXF1U3PRR823 6OyymG3eDpTouua5T1AUfnnZX4w6QQtWHJWPSvodH5Bv3DsC9vrfEbba2eAf8blJjjUi h8ng== MIME-Version: 1.0 Received: by 10.216.141.164 with SMTP id g36mr11032164wej.119.1338394450258; Wed, 30 May 2012 09:14:10 -0700 (PDT) Received: by 10.223.96.76 with HTTP; Wed, 30 May 2012 09:14:10 -0700 (PDT) In-Reply-To: References: Date: Wed, 30 May 2012 21:44:10 +0530 Message-ID: Subject: Re: Hive UDF error : numberformat exception (String to Integer) conversion From: praveenesh kumar To: user@hive.apache.org Content-Type: multipart/alternative; boundary=0016e6d99f1ac6ec7e04c1433dfd X-Virus-Checked: Checked by ClamAV on apache.org --0016e6d99f1ac6ec7e04c1433dfd Content-Type: text/plain; charset=ISO-8859-1 I have done both the things. There is no null issue here. Checked the nulls also. Sorry not mentioned in the code. I also have made a main function and called my evaluate function. If I am passing a string, its working fine. Problem is of numberformat exception. Integer.parseInt is throwing this.. I don't know why... I am converting hadoop's Text object to String.. Splitting it.. converting into String array.. and giving String inside is giving me problems. What could be the reason.. I know its not at all hive related..Its java mistake only. Please help me out in resolving this issue. Sounds embarassing, but don't know why I am not able to see the mistake I am doing. Regards, Praveenesh On Wed, May 30, 2012 at 8:17 PM, Nitin Pawar wrote: > i won't tell the error but I would recommend to write a main function in > your udf and try with sample inputs which you are expecting in your query. > > You will know whats the error you are committing > > > On Wed, May 30, 2012 at 8:14 PM, Edward Capriolo wrote: > >> You should to try catch and return NULL on bad data. The issue is if >> you have a single bad row the UDF will throw a exception up the chain. >> It will try again, it will fail again, ultimately the job will fail. >> >> On Wed, May 30, 2012 at 10:40 AM, praveenesh kumar >> wrote: >> > Hello Hive Users, >> > >> > There is a strange situation I am facing. >> > >> > I have a string column in my Hive table ( its IP address). I am >> creating a >> > UDF where I am taking this string column and converting it into Long >> value. >> > Its a simple UDF. Following is my code : >> > >> > package com.practice.hive.udf; >> > public class IPtoINT extends UDF { >> > public static LongWritable execute(Text addr) { >> > >> > String[] addrArray = addr.toString().split("\\."); >> > >> > long num = 0; >> > >> > for (int i=0;i> > int power = 3-i; >> > num += ((Integer.parseInt(addrArray[i])%256 * >> > Math.pow(256,power))); >> > } >> > return new LongWritable(num); >> > } >> > } >> > >> > After creating jar, I am running the following commands: >> > >> > $ hive >> > hive > add jar /home/hadoop/Desktop/HiveData/IPtoINT.jar; >> > hive > create temporary function ip2int as >> 'com.practice.hive.udf.IPtoINT'; >> > hive > select ip2int(ip1) from sample_data; >> > >> > But running the above, is giving me the following error: >> > >> > java.lang.RuntimeException: >> > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error >> while >> > processing row >> > {"ip1":"1.0.144.36","ip2":16814116,"country":"Thailand","key":null} >> > at >> > org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) >> > at >> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) >> > at >> > org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) >> > at >> org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) >> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >> > at java.security.AccessController.doPrivileged(Native >> > Method) >> > at javax.security.auth.Subject.doAs(Subject.java:415) >> > at >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) >> > at org.apache.hadoop.mapred.Child.main(Child.java:249) >> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive >> Runtime >> > Error while processing row >> > {"ip1":"1.0.144.36","ip2":16814116,"country":"Thailand","key":null} >> > at >> > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550) >> > at >> > org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) >> > ... 8 more >> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to >> > execute method public org.apache.hadoop.io.Text >> > com.musigma.hive.udf.ip2int.evaluate(org.apache.hadoop.io.Text) on >> object >> > com.musigma.hive.udf.ip2int@19a4d79 of class >> com.musigma.hive.udf.ip2int >> > with arguments {1.0.144.36:org.apache.hadoop.io.Text} of size 1 >> > at >> > >> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:848) >> > at >> > >> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:181) >> > at >> > >> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163) >> > at >> > >> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:76) >> > at >> > org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) >> > at >> > org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) >> > at >> > >> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) >> > at >> > org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) >> > at >> > org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) >> > at >> > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:531) >> > ... 9 more >> > Caused by: java.lang.reflect.InvocationTargetException >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > Method) >> > at >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> > at >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:601) >> > at >> > >> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:824) >> > ... 18 more >> > Caused by: java.lang.NumberFormatException: For input string: "1" >> > at >> > >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> > at java.lang.Long.parseLong(Long.java:441) >> > at java.lang.Long.(Long.java:702) >> > at com.musigma.hive.udf.ip2int.evaluate(ip2int.java:11) >> > ... 23 more >> > >> > >> > If I am running the HIVE UDF like --- select ip2int("102.134.123.1") >> from >> > sample_data; Its not giving any error. >> > Strange thing is, its numberformat exception. I am not able to use the >> > string into int in my java. Very strange issue. >> > >> > Can someone please tell me what stupid mistake I am doing ? >> > >> > Is there any other UDF that does string to int/long coversion ? >> > >> > Regards, >> > Praveenesh >> > > > > -- > Nitin Pawar > > --0016e6d99f1ac6ec7e04c1433dfd Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I have done both the things.
There is no null issue here. Checked the nu= lls also. Sorry not mentioned in the code.
I also have made a main funct= ion and called my evaluate function. If I am passing a string, its working = fine.

Problem is of numberformat exception.
Integer.parseInt is throwing t= his.. I don't know why... I am=A0 converting hadoop's Text object t= o String.. Splitting it.. converting into String array.. and giving String = inside is giving me problems. What could be the reason.. I know its not at = all hive related..Its java mistake only. Please help me out in resolving th= is issue. Sounds embarassing, but don't know why I am not able to see t= he mistake I am doing.

Regards,
Praveenesh



On Wed= , May 30, 2012 at 8:17 PM, Nitin Pawar <nitinpawar432@gmail.com&= gt; wrote:
i won't tell the erro= r but I would recommend to write a main function in your udf and try with s= ample inputs which you are expecting in your query.=A0

You will know whats the error you are committing=A0


On Wed, May 30, 2012 at 8:14 PM, Edward Capr= iolo <edlinuxguru@gmail.com> wrote:
You should to try catch and return NULL on bad data. The issue is if
you have a single bad row the UDF will throw a exception up the chain.
It will try again, it will fail again, ultimately the job will fail.

On Wed, May 30, 2012 at 10:40 AM, praveenesh kumar <praveenesh@gmail.com> wrote: > Hello Hive Users,
>
> There is a strange situation I am facing.
>
> I have a string column in my Hive table ( its IP address). I am creati= ng a
> UDF where I am taking this string column and converting it into Long v= alue.
> Its a simple UDF. Following is my code :
>
> package com.practice.hive.udf;
> public class IPtoINT extends UDF {
> public static LongWritable execute(Text addr) {
>
> =A0=A0=A0=A0 String[] addrArray =3D addr.toString().split("\\.&qu= ot;);
>
> =A0=A0=A0=A0 long num =3D 0;
>
> =A0=A0=A0=A0 for (int i=3D0;i<addrArray.length;i++) {
> =A0=A0=A0=A0=A0=A0=A0=A0=A0 int power =3D 3-i;
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 num +=3D ((Integer.parseInt(addrArra= y[i])%256 *
> Math.pow(256,power)));
> =A0=A0=A0=A0 }
> =A0=A0=A0=A0 return new LongWritable(num);
> =A0 }
> }
>
> After creating jar, I am running the following commands:
>
> $ hive
> hive > add jar /home/hadoop/Desktop/HiveData/IPtoINT.jar;
> hive > create temporary function ip2int as 'com.practice.hive.u= df.IPtoINT';
> hive > select ip2int(ip1) from sample_data;
>
> But running the above, is giving me the following error:
>
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error w= hile
> processing row
> {"ip1":"1.0.144.36","ip2":16814116,"= ;country":"Thailand","key":null}
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.map= red.MapRunner.run(MapRunner.java:50)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.map= red.MapTask.run(MapTask.java:372)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.map= red.Child$4.run(Child.java:255)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.security.AccessC= ontroller.doPrivileged(Native
> Method)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at javax.security.auth.S= ubject.doAs(Subject.java:415)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1059)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.map= red.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runt= ime
> Error while processing row
> {"ip1":"1.0.144.36","ip2":16814116,"= ;country":"Thailand","key":null}
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:55= 0)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to=
> execute method public org.apache.hadoop.io.Text
> com.musigma.hive.udf.ip2int.evaluate(org.apache.hadoop.io.Text)=A0 on = object
> com.musigma.hive.udf.ip2int@19a4d79 of class com.musigma.hive.udf.ip2i= nt
> with arguments {1.0.144.36:org.apache.hadoop.io.Text} of size 1
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistr= y.java:848)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(Generi= cUDFBridge.java:181)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(E= xprNodeGenericFuncEvaluator.java:163)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator= .java:76)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOp= erator.java:83)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:53= 1)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ... 9 more
> Caused by: java.lang.reflect.InvocationTargetException
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMet= hodAccessorImpl.invoke0(Native
> Method)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j= ava:57)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:43)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Met= hod.invoke(Method.java:601)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistr= y.java:824)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ... 18 more
> Caused by: java.lang.NumberFormatException: For input string: "1&= quot;
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at
> java.lang.NumberFormatException.forInputString(NumberFormatException.j= ava:65)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.lang.Long.parseL= ong(Long.java:441)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.lang.Long.<in= it>(Long.java:702)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at com.musigma.hive.udf.= ip2int.evaluate(ip2int.java:11)
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ... 23 more
>
>
> If I am running the HIVE UDF like --- select ip2int("102.134.123.= 1") from
> sample_data; Its not giving any error.
> Strange thing is, its numberformat exception. I am not able to use the=
> string into int in my java. Very strange issue.
>
> Can someone please tell me what stupid mistake I am doing ?
>
> Is there any other UDF that does string to int/long coversion ?
>
> Regards,
> Praveenesh



<= /div>--
Nitin Pawar
<= br>

--0016e6d99f1ac6ec7e04c1433dfd--