spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-18527) UDAFPercentile (bigint, array<double>) needs explicity cast to double
Date Mon, 28 Nov 2016 12:06:58 GMT

     [ https://issues.apache.org/jira/browse/SPARK-18527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-18527:
------------------------------------

    Assignee: Apache Spark

> UDAFPercentile (bigint, array<double>) needs explicity cast to double
> ---------------------------------------------------------------------
>
>                 Key: SPARK-18527
>                 URL: https://issues.apache.org/jira/browse/SPARK-18527
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.0, 2.0.1
>         Environment: spark-2.0.1-bin-hadoop2.7/bin/spark-shell
>            Reporter: Fabian Boehnlein
>            Assignee: Apache Spark
>
> Same bug as SPARK-16228 but 
> {code}_FUNC_(bigint, array<double>) {code}
> instead of 
> {code}_FUNC_(bigint, double){code}
> Fix of SPARK-16228 only fixes the non-array case that was hit.
> {code}
> sql("select percentile(value, array(0.5,0.99)) from values 1,2,3 T(value)")
> {code}
> fails in Spark 2 shell.
> Longer example
> {code}
> case class Record(key: Long, value: String)
> val recordsDF = spark.createDataFrame((1 to 100).map(i => Record(i.toLong, s"val_$i")))
> recordsDF.createOrReplaceTempView("records")
> sql("SELECT percentile(key, Array(0.95, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1))
AS test FROM records")
> org.apache.spark.sql.AnalysisException: No handler for Hive UDF 'org.apache.hadoop.hive.ql.udf.UDAFPercentile':
org.apache.hadoop.hive.ql.exec.NoMatchingMethodException: No matching method for class org.apache.had
> oop.hive.ql.udf.UDAFPercentile with (bigint, array<decimal(38,18)>). Possible choices:
_FUNC_(bigint, array<double>)  _FUNC_(bigint, double)  ; line 1 pos 7
>   at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getMethodInternal(FunctionRegistry.java:1164)
>   at org.apache.hadoop.hive.ql.exec.DefaultUDAFEvaluatorResolver.getEvaluatorClass(DefaultUDAFEvaluatorResolver.java:83)
>   at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge.getEvaluator(GenericUDAFBridge.java:56)
>   at org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:47){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message