spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwend...@apache.org
Subject [1/2] git commit: Adding disclaimer for shuffle file consolidation
Date Sat, 07 Dec 2013 04:16:30 GMT
Updated Branches:
  refs/heads/master e5d5728b7 -> 1b38f5f27


Adding disclaimer for shuffle file consolidation


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/b9451acd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/b9451acd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/b9451acd

Branch: refs/heads/master
Commit: b9451acdf4c73ab4b0b9d9827ab1ccedbcfb144f
Parents: 241336a
Author: Patrick Wendell <pwendell@gmail.com>
Authored: Fri Dec 6 19:25:28 2013 -0800
Committer: Patrick Wendell <pwendell@gmail.com>
Committed: Fri Dec 6 19:25:28 2013 -0800

----------------------------------------------------------------------
 docs/configuration.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b9451acd/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 22abe1c..65b41b4 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -331,7 +331,7 @@ Apart from these, the following properties are also available, and may
be useful
   <td>spark.shuffle.consolidateFiles</td>
   <td>false</td>
   <td>
-    If set to "true", consolidates intermediate files created during a shuffle. Creating
fewer files can improve filesystem performance if you run shuffles with large numbers of reduce
tasks.
+    If set to "true", consolidates intermediate files created during a shuffle. Creating
fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks.
It is reccomended to set this to "true" when using ext4 or xfs filesystems. On ext3, this
option might degrade performance on machines with many (>8) cores due to filesystem limitations.
   </td>
 </tr>
 <tr>


Mime
View raw message