spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Armbrust (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-6054) SQL UDF returning object of case class; regression from 1.2.0
Date Mon, 23 Mar 2015 23:18:53 GMT

     [ https://issues.apache.org/jira/browse/SPARK-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Armbrust reassigned SPARK-6054:
---------------------------------------

    Assignee: Michael Armbrust

> SQL UDF returning object of case class; regression from 1.2.0
> -------------------------------------------------------------
>
>                 Key: SPARK-6054
>                 URL: https://issues.apache.org/jira/browse/SPARK-6054
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.3.0
>         Environment: Windows 8, Scala 2.11.2, Spark 1.3.0 RC1
>            Reporter: Spiro Michaylov
>            Assignee: Michael Armbrust
>            Priority: Blocker
>
> The following code fails with a stack trace beginning with:
> {code}
> 15/02/26 23:21:32 ERROR Executor: Exception in task 2.0 in stage 7.0 (TID 422)
> org.apache.spark.sql.catalyst.errors.package$TreeNodeException: makeCopy, tree: scalaUDF(sales#2,discounts#3)
> 	at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:47)
> 	at org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:309)
> 	at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:237)
> 	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
> 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
> {code}
> Here is the 1.3.0 version of the code:
> {code}
> case class SalesDisc(sales: Double, discounts: Double)
> def makeStruct(sales: Double, disc:Double) = SalesDisc(sales, disc)
> sqlContext.udf.register("makeStruct", makeStruct _)
> val withStruct =
>       sqlContext.sql("SELECT id, sd.sales FROM (SELECT id, makeStruct(sales, discounts)
AS sd FROM customerTable) AS d")
>     withStruct.foreach(println)
> {code}
> This used to work in 1.2.0. Interestingly, the following simplified version fails similarly,
even though it seems to me to be VERY similar to the last test in the UDFSuite:
> {code}
> SELECT makeStruct(sales, discounts) AS sd FROM customerTable
> {code}
> The data table is defined thus:
> {code}
>   val custs = Seq(
>       Cust(1, "Widget Co", 120000.00, 0.00, "AZ"),
>       Cust(2, "Acme Widgets", 410500.00, 500.00, "CA"),
>       Cust(3, "Widgetry", 410500.00, 200.00, "CA"),
>       Cust(4, "Widgets R Us", 410500.00, 0.0, "CA"),
>       Cust(5, "Ye Olde Widgete", 500.00, 0.0, "MA")
>     )
>     val customerTable = sc.parallelize(custs, 4).toDF()
>     customerTable.registerTempTable("customerTable")
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message