spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From redsanket <...@git.apache.org>
Subject [GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Date Tue, 18 Sep 2018 19:12:33 GMT
Github user redsanket commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22173#discussion_r218559203
  
    --- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
---
    @@ -281,4 +282,31 @@ public Properties cryptoConf() {
       public long maxChunksBeingTransferred() {
         return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
       }
    +
    +  /**
    +   * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
    +   * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
    +   * Although when calling the async writeAndFlush on the underlying channel to send
    +   * response back to client, the I/O on the channel is still being handled by
    +   * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
    +   * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
    +   * threads for the completion of sending back responses, we are able to put a limit
on
    +   * the max number of threads from TransportServer's default EventLoopGroup that are
    +   * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
    +   * and could take long time to process due to disk contentions. By configuring a slightly
    +   * higher number of shuffler server threads, we are able to reserve some threads for
    +   * handling other RPC messages, thus making the Client less likely to experience timeout
    +   * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
    +   * or io.serverThreads. 90 would mean 90% of 2*#cores or 90% of io.serverThreads
    +   * which equals 0.9 * 2*#cores or 0.9 * io.serverThreads.
    +   */
    +  public int chunkFetchHandlerThreads() {
    +    if (!this.getModuleName().equalsIgnoreCase("shuffle")) {
    +      return 0;
    +    }
    +    int chunkFetchHandlerThreadsPercent =
    +            conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
    --- End diff --
    
    Yes it is documented above... if it is 0 or 100 it is 2*#cores or io.serverThreads


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message