spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wenc...@apache.org
Subject spark git commit: [SPARK-21530] Update description of spark.shuffle.maxChunksBeingTransferred.
Date Thu, 27 Jul 2017 03:55:52 GMT
Repository: spark
Updated Branches:
  refs/heads/master 60472dbfd -> cfb25b27c


[SPARK-21530] Update description of spark.shuffle.maxChunksBeingTransferred.

## What changes were proposed in this pull request?

Update the description of `spark.shuffle.maxChunksBeingTransferred` to include that the new
coming connections will be closed when the max is hit and client should have retry mechanism.

Author: jinxing <jinxing6042@126.com>

Closes #18735 from jinxing64/SPARK-21530.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cfb25b27
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cfb25b27
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cfb25b27

Branch: refs/heads/master
Commit: cfb25b27c0b32a8a70a518955fb269314b1fd716
Parents: 60472db
Author: jinxing <jinxing6042@126.com>
Authored: Thu Jul 27 11:55:48 2017 +0800
Committer: Wenchen Fan <wenchen@databricks.com>
Committed: Thu Jul 27 11:55:48 2017 +0800

----------------------------------------------------------------------
 .../main/java/org/apache/spark/network/util/TransportConf.java | 6 +++++-
 docs/configuration.md                                          | 6 +++++-
 2 files changed, 10 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/cfb25b27/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
----------------------------------------------------------------------
diff --git a/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
b/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
index ea52e9f..88256b8 100644
--- a/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
+++ b/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
@@ -258,7 +258,11 @@ public class TransportConf {
   }
 
   /**
-   * The max number of chunks allowed to being transferred at the same time on shuffle service.
+   * The max number of chunks allowed to be transferred at the same time on shuffle service.
+   * Note that new incoming connections will be closed when the max number is hit. The client
will
+   * retry according to the shuffle retry configs (see `spark.shuffle.io.maxRetries` and
+   * `spark.shuffle.io.retryWait`), if those limits are reached the task will fail with fetch
+   * failure.
    */
   public long maxChunksBeingTransferred() {
     return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);

http://git-wip-us.apache.org/repos/asf/spark/blob/cfb25b27/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index f4b6f46..500f980 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -635,7 +635,11 @@ Apart from these, the following properties are also available, and may
be useful
   <td><code>spark.shuffle.maxChunksBeingTransferred</code></td>
   <td>Long.MAX_VALUE</td>
   <td>
-    The max number of chunks allowed to being transferred at the same time on shuffle service.
+    The max number of chunks allowed to be transferred at the same time on shuffle service.
+    Note that new incoming connections will be closed when the max number is hit. The client
will
+    retry according to the shuffle retry configs (see <code>spark.shuffle.io.maxRetries</code>
and
+    <code>spark.shuffle.io.retryWait</code>), if those limits are reached the
task will fail with
+    fetch failure.
   </td>
 </tr>
 <tr>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message