spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-21041) With whole-stage codegen, SparkSession.range()'s behavior is inconsistent with SparkContext.range()
Date Fri, 09 Jun 2017 22:38:18 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045146#comment-16045146
] 

Dongjoon Hyun commented on SPARK-21041:
---------------------------------------

Oh, interesting. I'll take a look. [~rednaxelafx].

> With whole-stage codegen, SparkSession.range()'s behavior is inconsistent with SparkContext.range()
> ---------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-21041
>                 URL: https://issues.apache.org/jira/browse/SPARK-21041
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Kris Mok
>
> When whole-stage codegen is enabled, in face of integer overflow, SparkSession.range()'s
behavior is inconsistent with when codegen is turned off, while the latter is consistent with
SparkContext.range()'s behavior.
> The following Spark Shell session shows the inconsistency:
> {code:java}
> scala> sc.range
>    def range(start: Long,end: Long,step: Long,numSlices: Int): org.apache.spark.rdd.RDD[Long]
> scala> spark.range
>                                                                                     
                
> def range(start: Long,end: Long,step: Long,numPartitions: Int): org.apache.spark.sql.Dataset[Long]
  
> def range(start: Long,end: Long,step: Long): org.apache.spark.sql.Dataset[Long]     
                
> def range(start: Long,end: Long): org.apache.spark.sql.Dataset[Long]                
                
> def range(end: Long): org.apache.spark.sql.Dataset[Long] 
> scala> sc.range(java.lang.Long.MAX_VALUE - 3, java.lang.Long.MIN_VALUE + 2, 1).collect
> res1: Array[Long] = Array()
> scala> spark.range(java.lang.Long.MAX_VALUE - 3, java.lang.Long.MIN_VALUE + 2, 1).collect
> res2: Array[Long] = Array(9223372036854775804, 9223372036854775805, 9223372036854775806)
> scala> spark.conf.set("spark.sql.codegen.wholeStage", false)
> scala> spark.range(java.lang.Long.MAX_VALUE - 3, java.lang.Long.MIN_VALUE + 2, 1).collect
> res5: Array[Long] = Array()
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message