spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwend...@apache.org
Subject [1/2] git commit: Enable shuffle consolidation by default.
Date Fri, 10 Jan 2014 07:38:12 GMT
Updated Branches:
  refs/heads/master 997c830e0 -> dd03cea02


Enable shuffle consolidation by default.

Bump this to being enabled for 0.9.0.


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/460f655c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/460f655c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/460f655c

Branch: refs/heads/master
Commit: 460f655cc60b1a1759b6f85ae8860595fc7586b4
Parents: 4b074fa
Author: Patrick Wendell <pwendell@gmail.com>
Authored: Thu Jan 9 22:42:50 2014 -0800
Committer: Patrick Wendell <pwendell@gmail.com>
Committed: Thu Jan 9 22:42:50 2014 -0800

----------------------------------------------------------------------
 .../main/scala/org/apache/spark/storage/ShuffleBlockManager.scala  | 2 +-
 docs/configuration.md                                              | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/460f655c/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala
index e2b2429..6e0ff14 100644
--- a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala
+++ b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala
@@ -64,7 +64,7 @@ class ShuffleBlockManager(blockManager: BlockManager) {
   // Turning off shuffle file consolidation causes all shuffle Blocks to get their own file.
   // TODO: Remove this once the shuffle file consolidation feature is stable.
   val consolidateShuffleFiles =
-    conf.getBoolean("spark.shuffle.consolidateFiles", false)
+    conf.getBoolean("spark.shuffle.consolidateFiles", true)
 
   private val bufferSize = conf.getInt("spark.shuffle.file.buffer.kb", 100) * 1024
 

http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/460f655c/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 6717757..b1a0e19 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -371,7 +371,7 @@ Apart from these, the following properties are also available, and may
be useful
 
 <tr>
   <td>spark.shuffle.consolidateFiles</td>
-  <td>false</td>
+  <td>true</td>
   <td>
     If set to "true", consolidates intermediate files created during a shuffle. Creating
fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks.
It is recommended to set this to "true" when using ext4 or xfs filesystems. On ext3, this
option might degrade performance on machines with many (>8) cores due to filesystem limitations.
   </td>


Mime
View raw message