Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 9 Aug 2016 23:36:20 +0000 (UTC)
From: "Shivaram Venkataraman (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12994735.1470264830000.266903.1470785780454@Atlassian.JIRA>
In-Reply-To: <JIRA.12994735.1470264830000@Atlassian.JIRA>
References: <JIRA.12994735.1470264830000@Atlassian.JIRA> <JIRA.12994735.1470264830785@arcas>
Subject: [jira] [Commented] (SPARK-16883) SQL decimal type is not properly
 cast to number when collecting SparkDataFrame
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 09 Aug 2016 23:36:22 -0000


    [ https://issues.apache.org/jira/browse/SPARK-16883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414446#comment-15414446 ] 

Shivaram Venkataraman commented on SPARK-16883:
-----------------------------------------------

Yeah I see that - The change I am proposing is to add another case inside writeObject to convert DecimalTypes to doubles using something like the function in https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L344

I guess the question is whether this change is more intrusive than the other one and how will this impact handling more types in the future. It'll be great if you can try out one or both of them and open a PR for more discussion.

> SQL decimal type is not properly cast to number when collecting SparkDataFrame
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-16883
>                 URL: https://issues.apache.org/jira/browse/SPARK-16883
>             Project: Spark
>          Issue Type: Bug
>          Components: SparkR
>    Affects Versions: 2.0.0
>            Reporter: Hossein Falaki
>
> To reproduce run following code. As you can see "y" is a list of values.
> {code}
> registerTempTable(createDataFrame(iris), "iris")
> str(collect(sql("select cast('1' as double) as x, cast('2' as decimal) as y  from iris limit 5")))
> 'data.frame':	5 obs. of  2 variables:
>  $ x: num  1 1 1 1 1
>  $ y:List of 5
>   ..$ : num 2
>   ..$ : num 2
>   ..$ : num 2
>   ..$ : num 2
>   ..$ : num 2
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org