spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maropu <...@git.apache.org>
Subject [GitHub] spark pull request #19401: [SPARK-22176][SQL] Fix overflow issue in Dataset....
Date Sat, 30 Sep 2017 10:03:39 GMT
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19401#discussion_r142003100
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -238,8 +238,9 @@ class Dataset[T] private[sql](
       private[sql] def showString(
           _numRows: Int, truncate: Int = 20, vertical: Boolean = false): String = {
         val numRows = _numRows.max(0)
    -    val takeResult = toDF().take(numRows + 1)
    -    val hasMoreData = takeResult.length > numRows
    +    val numTotalRows = toDF().count()
    --- End diff --
    
    In the suggested, `hasMoreData` gets meaningless, so how about this?;
    ```
        val (data, hasMoreData) = if (numRows < Int.MaxValue) {
          val takeResult = toDF().take(numRows + 1)
          (takeResult.take(numRows), takeResult.length > numRows)
        } else {
          val takeResult = toDF().take(numRows)
          val numTotalRows = toDF().count()
          (takeResult.take(numRows), numTotalRows > numRows)
        }
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message