spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cloud-fan <...@git.apache.org>
Subject [GitHub] spark pull request #19788: [SPARK-9853][Core] Optimize shuffle fetch of cont...
Date Thu, 22 Feb 2018 07:33:55 GMT
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19788#discussion_r169873362
  
    --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
    @@ -280,6 +282,16 @@ private[spark] abstract class MapOutputTracker(conf: SparkConf) extends
Logging
         }
       }
     
    +  protected def supportsContinuousBlockBatchFetch(serializer: Serializer): Boolean =
{
    +    if (serializer == null || !serializer.supportsRelocationOfSerializedObjects) {
    +      return false
    +    }
    +    val compressionEnabled: Boolean = conf.getBoolean("spark.shuffle.compress", true)
    +    val compressionCodec: CompressionCodec = CompressionCodec.createCodec(conf)
    --- End diff --
    
    We only need to create the codec if `compressionEnabled` is true.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message