Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm
Precedence: bulk
From: viirya <git@git.apache.org>
To: reviews@spark.apache.org
Reply-To: reviews@spark.apache.org
References: <git-pr-19763-spark@git.apache.org>
In-Reply-To: <git-pr-19763-spark@git.apache.org>
Subject: [GitHub] spark pull request #19763: [SPARK-22537][core] Aggregation of map output sta...
Content-Type: text/plain
Message-Id: <20171124082000.AA14FDFDFB@git1-us-west.apache.org>
Date: Fri, 24 Nov 2017 08:20:00 +0000 (UTC)
archived-at: Fri, 24 Nov 2017 08:20:03 -0000

Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19763#discussion_r152914613
  
    --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
    @@ -485,4 +485,13 @@ package object config {
             "array in the sorter.")
           .intConf
           .createWithDefault(Integer.MAX_VALUE)
    +
    +  private[spark] val SHUFFLE_MAP_OUTPUT_PARALLEL_AGGREGATION_THRESHOLD =
    +    ConfigBuilder("spark.shuffle.mapOutput.parallelAggregationThreshold")
    +      .internal()
    +      .doc("Multi-thread is used when the number of mappers * shuffle partitions is greater than " +
    +        "or equal to this threshold.")
    --- End diff --
    
    After rethinking about this, I think it is better to indicate this threshold also determines the number of threads in parallelism. So it should not be set to zero or negative number.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org