Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DBB33D187 for ; Sun, 10 Feb 2013 18:36:45 +0000 (UTC) Received: (qmail 11510 invoked by uid 500); 10 Feb 2013 18:36:44 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 11415 invoked by uid 500); 10 Feb 2013 18:36:43 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 11404 invoked by uid 99); 10 Feb 2013 18:36:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Feb 2013 18:36:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of grover.markgrover@gmail.com designates 209.85.212.48 as permitted sender) Received: from [209.85.212.48] (HELO mail-vb0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Feb 2013 18:36:37 +0000 Received: by mail-vb0-f48.google.com with SMTP id fc21so3281677vbb.7 for ; Sun, 10 Feb 2013 10:36:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=s5c2bzMb6C+40jRJxWHdOL3fsDc8o0QkqGcljPZKMn0=; b=AThkmtFeeUZTL73oF6R8EVelRNjjvqd4V8h22ydNJ1qPHqBPQs/AxQ1SflkVUMDrn0 /1k3ARrLpf6GfsFTrIJR8jswxM2zhyhU7afVIbZp8K7Cpzzj7RjTqryOmQc/CuUZRWbU y9DewMztEy6DN9T/2hWWGNfQaCT6QSWcrE/NU9viG1LGDj80w36yRttaNNAx4w7AGCNK etJZxlbnVUehlJgxu3WFmcofdvF71qG/gWmwngS0/HZwepFULBf9+tzoaKdEVKfGi3V5 SFdi76UAWvfHkIMJ1uUOQNoPAWU5alnnJNMazbs7OdciDy0ZE6zAxuNJpJ2a4TLVeIXX 6ZDw== MIME-Version: 1.0 X-Received: by 10.58.205.179 with SMTP id lh19mr15627342vec.7.1360521375692; Sun, 10 Feb 2013 10:36:15 -0800 (PST) Received: by 10.58.137.37 with HTTP; Sun, 10 Feb 2013 10:36:15 -0800 (PST) In-Reply-To: References: Date: Sun, 10 Feb 2013 10:36:15 -0800 Message-ID: Subject: Re: Help to solve UDAF errors! From: Mark Grover To: user@hive.apache.org Content-Type: multipart/alternative; boundary=089e011841a04eb58b04d56311be X-Virus-Checked: Checked by ClamAV on apache.org --089e011841a04eb58b04d56311be Content-Type: text/plain; charset=ISO-8859-1 Hi Abhishek, The code looks incomplete. See the comment at https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java#L22 Those are all the methods your UDAF class needs to implement but you seem to be missing them. Mark On Sat, Feb 9, 2013 at 11:08 PM, Abhishek Bhattacharya wrote: > Thanks for the response. > The link to the code is: > https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java > Please let me know to fix it! > > Thanks, > Abhishek > > > > On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover wrote: > >> Abhishek, >> The code doesn't seem to be complete. >> >> Look at >> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.javafor reference. It has two terminate()'s - one for UDAF and one for the >> Evaluator. >> >> Do you mind posting your complete code on github somewhere so it's easier >> to analyze? >> >> Mark >> >> On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya wrote: >> >>> Hi, >>> >>> I have implemented a simple UDAF for top-n-percent as follows: >>> import java.util.ArrayList; >>> import java.util.Collections; >>> >>> import org.apache.hadoop.hive.ql.exec.UDAF; >>> import org.apache.hadoop.hive.ql.exec.UDAFEvaluator; >>> >>> public class UDAFTopNPercent extends UDAF{ >>> >>> public static class Result { >>> ArrayList list; >>> double min; >>> } >>> >>> public class TopNPercentEvaluator implements UDAFEvaluator { >>> >>> private Result res; >>> private int rowIndex; >>> private int percent; >>> >>> public TopNPercentEvaluator() { >>> super(); >>> res = new Result(); >>> init(); >>> rowIndex = 0; >>> } >>> @Override >>> public void init() { >>> res.list = new ArrayList(); >>> res.min = Double.MAX_VALUE; >>> } >>> >>> public boolean iterate(Double rowVal, int pct) { >>> ArrayList resList = res.list; >>> rowIndex++; >>> resList.add(rowVal); >>> percent = pct; >>> return true; >>> } >>> >>> public ArrayList terminatePartial() { >>> ArrayList resList = res.list; >>> Collections.sort(resList); >>> return resList; >>> } >>> >>> public boolean merge(ArrayList otherList) { >>> ArrayList resList = res.list; >>> resList.addAll(otherList); >>> return true; >>> } >>> >>> public ArrayList terminate() { >>> ArrayList resList = res.list; >>> double num_rows = (double)percent/100.0*rowIndex; >>> Collections.sort(resList); >>> int lastIdx = resList.size()- (int) num_rows; >>> if(lastIdx <= 0) { >>> return resList; >>> } >>> for(int i=0; i>> resList.remove(i); >>> } >>> return resList; >>> } >>> } >>> >>> /** >>> * @param args >>> */ >>> public static void main(String[] args) { >>> // TODO Auto-generated method stub >>> >>> } >>> >>> } >>> >>> But throws some error such as first few lines are: >>> FAILED: Hive Internal Error: >>> java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector >>> cannot be cast to >>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector) >>> java.lang.ClassCastException: >>> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector >>> cannot be cast to >>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector >>> at >>> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116) >>> at >>> org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.(GenericUDFUtils.java:300) >>> at >>> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129) >>> >>> Please help me to debug this! >>> Is it throwing from returning ArrayList in terminate()? >>> How should I return a List from UDAF? >>> >>> Thanks, >>> Abhishek >>> >> >> > > > -- > Thanks and Regards, > > Abhishek Bhattacharya > PhD Computer Science > School of Computing and Information Sciences > Florida International University > --089e011841a04eb58b04d56311be Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Abhishek,
The code looks incomplete.

Those are all the methods your UDAF class needs to implement but you s= eem to be missing them.

Mark

On Sat, Feb 9, 2013 at 11:08 PM, Abhishek Bhattacharya <abhat0= 02@fiu.edu> wrote:
Thanks for the response.The link to the code is:=A0https://github.co= m/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
Please let me know to fix it!

Thanks,
Abhishek



On Fri, Feb 8, 2013 at 5:02= PM, Mark Grover <grover.markgrover@gmail.com> wro= te:
Abhishek,
The code doesn't seem to b= e complete.

Look at=A0https://github.com/apache/hive/blob/trunk/ql= /src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.java for referen= ce. It has two terminate()'s - one for UDAF and one for the Evaluator.<= /div>

Do you mind posting your complete code on github somewh= ere so it's easier to analyze?

Mark
=
On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bha= ttacharya <abhat002@fiu.edu> wrote:
Hi,

I have implemented a simple UDA= F for top-n-percent as follows:
import java.util.ArrayList;
import java.util.Collections;

import org.apache.hadoop.hive.ql.exec.UDAF;
import org.apache.hadoop.hiv= e.ql.exec.UDAFEvaluator;

public class UDAFTopNPercent extends UDAF{<= br>=A0=A0=A0=A0
=A0=A0=A0 public static class Result {
=A0=A0=A0 =A0= =A0=A0 ArrayList<Double> list;
=A0=A0=A0 =A0=A0=A0 double min;
=A0=A0=A0 }
=A0=A0=A0=A0
=A0=A0=A0= public class TopNPercentEvaluator implements UDAFEvaluator {
=A0=A0=A0 = =A0=A0=A0=A0
=A0=A0=A0 =A0=A0=A0 private Result res;
=A0=A0=A0 =A0=A0= =A0 private int rowIndex;
=A0=A0=A0 =A0=A0=A0 private int percent;
= =A0=A0=A0 =A0=A0=A0=A0
=A0=A0=A0 =A0=A0=A0 public TopNPercentEvaluator() {
=A0=A0=A0 =A0=A0=A0 = =A0=A0=A0 super();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 res =3D new Result();=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 init();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 ro= wIndex =3D 0;
=A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0 @Override
= =A0=A0=A0 =A0=A0=A0 public void init() {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 res.list =3D new ArrayList<Double>(); =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 res.min =3D Double.MAX_VALUE;
=A0=A0=A0 = =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0=A0
=A0=A0=A0 =A0=A0=A0 public boolean= iterate(Double rowVal, int pct) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 ArrayLi= st<Double> resList =3D res.list;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 row= Index++;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 resList.add(rowVal);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 percent =3D pct;
=A0=A0=A0 =A0=A0=A0 =A0= =A0=A0 return true;
=A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0=A0
= =A0=A0=A0 =A0=A0=A0 public ArrayList<Double> terminatePartial() {
= =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 ArrayList<Double> resList =3D res.list;=
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 Collections.sort(resList);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return resList;
=A0=A0=A0 =A0=A0=A0 }
= =A0=A0=A0 =A0=A0=A0=A0
=A0=A0=A0 =A0=A0=A0 public boolean merge(ArrayLis= t<Double> otherList) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 ArrayList<= Double> resList =3D res.list;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 resList.a= ddAll(otherList);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return true;
=A0=A0=A0 =A0=A0=A0 }
=A0= =A0=A0 =A0=A0=A0=A0
=A0=A0=A0 =A0=A0=A0 public ArrayList<Double> t= erminate() {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 ArrayList<Double> resLi= st =3D res.list;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 double num_rows =3D (doub= le)percent/100.0*rowIndex;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 Collections.sort(resList);
=A0=A0=A0 =A0= =A0=A0 =A0=A0=A0 int lastIdx =3D resList.size()- (int) num_rows;
=A0=A0= =A0 =A0=A0=A0 =A0=A0=A0 if(lastIdx <=3D 0) {
=A0=A0=A0 =A0=A0=A0 =A0= =A0=A0 =A0=A0=A0 return resList;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
=A0= =A0=A0 =A0=A0=A0 =A0=A0=A0 for(int i=3D0; i<lastIdx; i++) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 resList.remove(i);
=A0=A0=A0 =A0= =A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return resList;
=A0= =A0=A0 =A0=A0=A0 }
=A0=A0=A0 }

=A0=A0=A0 /**
=A0=A0=A0 =A0* @p= aram args
=A0=A0=A0 =A0*/
=A0=A0=A0 public static void main(String[] = args) {
=A0=A0=A0 =A0=A0=A0 // TODO Auto-generated method stub

=A0=A0=A0 }

}

But throws some error such as fi= rst few lines are:
FAILED: Hive Internal Error: java.lang.ClassCastExcep= tion(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatO= bjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspec= tor.StructObjectInspector)
java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector= .primitive.WritableFloatObjectInspector cannot be cast to org.apache.hadoop= .hive.serde2.objectinspector.StructObjectInspector
=A0=A0=A0=A0=A0=A0=A0= at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters= .getConverter(ObjectInspectorConverters.java:116)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.udf.generic.GenericUDFUt= ils$ConversionHelper.<init>(GenericUDFUtils.java:300)
=A0=A0=A0=A0= =A0=A0=A0 at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$Generi= cUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129)

Please help me to debug this!
Is it throwing f= rom returning ArrayList<Double> in terminate()?
How sho= uld I return a List from UDAF?

Thanks,
Abhishek




--
Thanks and Regards,

= Abhishek Bhattacharya
PhD Computer Science
School of Computing and In= formation Sciences
Florida International University

--089e011841a04eb58b04d56311be--