Mailing-List: contact commits-help@spark.apache.org; run by ezmlm
Precedence: bulk
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: wenchen@apache.org
To: commits@spark.apache.org
Message-Id: <9f2ca28788be43f9934beee0fe44e0de@git.apache.org>
Subject: spark git commit: [SPARK-21343] Refine the document for
 spark.reducer.maxReqSizeShuffleToMem.
Date: Sat,  8 Jul 2017 16:28:22 +0000 (UTC)
archived-at: Sat, 08 Jul 2017 16:28:24 -0000

Repository: spark
Updated Branches:
  refs/heads/branch-2.2 c8d7855b9 -> 964332b28


[SPARK-21343] Refine the document for spark.reducer.maxReqSizeShuffleToMem.

## What changes were proposed in this pull request?

In current code, reducer can break the old shuffle service when `spark.reducer.maxReqSizeShuffleToMem` is enabled. Let's refine document.

Author: jinxing <jinxing6042@126.com>

Closes #18566 from jinxing64/SPARK-21343.

(cherry picked from commit 062c336d06a0bd4e740a18d2349e03e311509243)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/964332b2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/964332b2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/964332b2

Branch: refs/heads/branch-2.2
Commit: 964332b2879af048a95606dfcb4f2cb2e356135b
Parents: c8d7855
Author: jinxing <jinxing6042@126.com>
Authored: Sun Jul 9 00:27:58 2017 +0800
Committer: Wenchen Fan <wenchen@databricks.com>
Committed: Sun Jul 9 00:28:14 2017 +0800

----------------------------------------------------------------------
 .../scala/org/apache/spark/internal/config/package.scala  |  6 ++++--
 docs/configuration.md                                     | 10 ++++++++++
 2 files changed, 14 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/964332b2/core/src/main/scala/org/apache/spark/internal/config/package.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 70feea8..1774673 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -289,9 +289,11 @@ package object config {
 
   private[spark] val REDUCER_MAX_REQ_SIZE_SHUFFLE_TO_MEM =
     ConfigBuilder("spark.reducer.maxReqSizeShuffleToMem")
-      .internal()
       .doc("The blocks of a shuffle request will be fetched to disk when size of the request is " +
-        "above this threshold. This is to avoid a giant request takes too much memory.")
+        "above this threshold. This is to avoid a giant request takes too much memory. We can " +
+        "enable this config by setting a specific value(e.g. 200m). Note that this config can " +
+        "be enabled only when the shuffle shuffle service is newer than Spark-2.2 or the shuffle" +
+        " service is disabled.")
       .bytesConf(ByteUnit.BYTE)
       .createWithDefault(Long.MaxValue)
 }

http://git-wip-us.apache.org/repos/asf/spark/blob/964332b2/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 459e944..f47967c 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -520,6 +520,16 @@ Apart from these, the following properties are also available, and may be useful
   </td>
 </tr>
 <tr>
+  <td><code>spark.reducer.maxReqSizeShuffleToMem</code></td>
+  <td>Long.MaxValue</td>
+  <td>
+    The blocks of a shuffle request will be fetched to disk when size of the request is above
+    this threshold. This is to avoid a giant request takes too much memory. We can enable this
+    config by setting a specific value(e.g. 200m). Note that this config can be enabled only when
+    the shuffle shuffle service is newer than Spark-2.2 or the shuffle service is disabled.
+  </td>
+</tr>
+<tr>
   <td><code>spark.shuffle.compress</code></td>
   <td>true</td>
   <td>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org