spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 10110346 <...@git.apache.org>
Subject [GitHub] spark pull request #20576: [SPARK-23389][CORE]When the shuffle dependency sp...
Date Thu, 01 Mar 2018 03:07:45 GMT
Github user 10110346 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20576#discussion_r171452995
  
    --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/SortShuffleManagerSuite.scala
---
    @@ -85,6 +85,14 @@ class SortShuffleManagerSuite extends SparkFunSuite with Matchers {
           mapSideCombine = false
         )))
     
    +    // We support serialized shuffle if we do not need to do map-side aggregation
    +    assert(canUseSerializedShuffle(shuffleDep(
    +      partitioner = new HashPartitioner(2),
    +      serializer = kryo,
    +      keyOrdering = None,
    +      aggregator = Some(mock(classOf[Aggregator[Any, Any, Any]])),
    +      mapSideCombine = false
    --- End diff --
    
    You can see this code: `def groupByKey(partitioner: Partitioner): RDD[(K, Iterable[V])]`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message