spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravindra Pesala (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-2693) Support for UDAF Hive Aggregates like PERCENTILE
Date Tue, 26 Aug 2014 14:05:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110707#comment-14110707
] 

Ravindra Pesala edited comment on SPARK-2693 at 8/26/14 2:05 PM:
-----------------------------------------------------------------

UDAF is deprecated in HIve, Though there can be few functions which could have implemented
using this interface. We can support the same in spark for backward compatability. 
As you mentioned supporting UDAF in spark requires to write a wrapper.

Please assign it to me.


was (Author: ravipesala):
UDAF is deprecated in HIve, Though there can be few functions which could have implemented
using this interface. We can support the same in spark for backward compatability. 
As you mentioned supporting UDAF in spark requires to write a wrapper.
*Please assign it to me.*

> Support for UDAF Hive Aggregates like PERCENTILE
> ------------------------------------------------
>
>                 Key: SPARK-2693
>                 URL: https://issues.apache.org/jira/browse/SPARK-2693
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Michael Armbrust
>
> {code}
> SELECT MIN(field1), MAX(field2), AVG(field3), PERCENTILE(field4), year,month,day FROM
 raw_data_table  GROUP BY year, month, day
> MIN, MAX and AVG functions work fine for me, but with PERCENTILE, I get an error as shown
below.
> Exception in thread "main" java.lang.RuntimeException: No handler for udf class org.apache.hadoop.hive.ql.udf.UDAFPercentile
>         at scala.sys.package$.error(package.scala:27)
>         at org.apache.spark.sql.hive.HiveFunctionRegistry$.lookupFunction(hiveUdfs.scala:69)
>         at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$4$$anonfun$applyOrElse$3.applyOrElse(Analyzer.scala:115)
>         at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$4$$anonfun$applyOrElse$3.applyOrElse(Analyzer.scala:113)
>         at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:165)
> {code}
> This aggregate extends UDAF, which we don't yet have a wrapper for.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message