spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From viirya <...@git.apache.org>
Subject [GitHub] spark pull request #19763: [SPARK-22537][core] Aggregation of map output sta...
Date Fri, 24 Nov 2017 08:20:00 GMT
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19763#discussion_r152914613
  
    --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
    @@ -485,4 +485,13 @@ package object config {
             "array in the sorter.")
           .intConf
           .createWithDefault(Integer.MAX_VALUE)
    +
    +  private[spark] val SHUFFLE_MAP_OUTPUT_PARALLEL_AGGREGATION_THRESHOLD =
    +    ConfigBuilder("spark.shuffle.mapOutput.parallelAggregationThreshold")
    +      .internal()
    +      .doc("Multi-thread is used when the number of mappers * shuffle partitions is greater
than " +
    +        "or equal to this threshold.")
    --- End diff --
    
    After rethinking about this, I think it is better to indicate this threshold also determines
the number of threads in parallelism. So it should not be set to zero or negative number.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message