spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jiangxb1987 <...@git.apache.org>
Subject [GitHub] spark pull request #19464: [SPARK-22233] [core] Allow user to filter out emp...
Date Wed, 11 Oct 2017 14:38:15 GMT
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19464#discussion_r144030461
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
    @@ -196,7 +196,10 @@ class HadoopRDD[K, V](
         // add the credentials here as this can be called before SparkContext initialized
         SparkHadoopUtil.get.addCredentials(jobConf)
         val inputFormat = getInputFormat(jobConf)
    -    val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
    +    var inputSplits = inputFormat.getSplits(jobConf, minPartitions)
    --- End diff --
    
    How about:
    ```
    val inputSplits = if (......) {
        inputFormat.getSplits(jobConf, minPartitions).filter(_.getLength > 0)
    } else {
       inputFormat.getSplits(jobConf, minPartitions)
    }
    ```
    We should alway try to not use `var`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message