spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From joshro...@apache.org
Subject spark git commit: [SPARK-6313] Add config option to disable file locks/fetchFile cache to ...
Date Tue, 17 Mar 2015 16:34:03 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.2 9ebd6f12e -> a2a94a154


[SPARK-6313] Add config option to disable file locks/fetchFile cache to ...

...support NFS mounts.

This is a work around for now with the goal to find a more permanent solution.
https://issues.apache.org/jira/browse/SPARK-6313

Author: nemccarthy <nathan@nemccarthy.me>

Closes #5036 from nemccarthy/master and squashes the following commits:

2eaaf42 [nemccarthy] [SPARK-6313] Update config wording doc for spark.files.useFetchCache
5de7eb4 [nemccarthy] [SPARK-6313] Add config option to disable file locks/fetchFile cache
to support NFS mounts

(cherry picked from commit 4cca3917dc30ee907e6cbd6a569b6ac58af963f7)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a2a94a15
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a2a94a15
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a2a94a15

Branch: refs/heads/branch-1.2
Commit: a2a94a154bdd00753b8d5e344d712664c7151050
Parents: 9ebd6f1
Author: nemccarthy <nathan@nemccarthy.me>
Authored: Tue Mar 17 09:33:11 2015 -0700
Committer: Josh Rosen <joshrosen@databricks.com>
Committed: Tue Mar 17 09:33:45 2015 -0700

----------------------------------------------------------------------
 core/src/main/scala/org/apache/spark/util/Utils.scala |  3 ++-
 docs/configuration.md                                 | 12 ++++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/a2a94a15/core/src/main/scala/org/apache/spark/util/Utils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala
index b92de55..7a3b674 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -398,7 +398,8 @@ private[spark] object Utils extends Logging {
       useCache: Boolean) {
     val fileName = url.split("/").last
     val targetFile = new File(targetDir, fileName)
-    if (useCache) {
+    val fetchCacheEnabled = conf.getBoolean("spark.files.useFetchCache", defaultValue = true)
+    if (useCache && fetchCacheEnabled) {
       val cachedFileName = s"${url.hashCode}${timestamp}_cache"
       val lockFileName = s"${url.hashCode}${timestamp}_lock"
       val localDir = new File(getLocalDir(conf))

http://git-wip-us.apache.org/repos/asf/spark/blob/a2a94a15/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index f8549df..d00c891 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -655,6 +655,18 @@ Apart from these, the following properties are also available, and may
be useful
   </td>
 </tr>
 <tr>
+  <td><code>spark.files.useFetchCache</code></td>
+  <td>true</td>
+  <td>
+    If set to true (default), file fetching will use a local cache that is shared by executors
+    that belong to the same application, which can improve task launching performance when
+    running many executors on the same host. If set to false, these caching optimizations
will
+    be disabled and all executors will fetch their own copies of files. This optimization
may be
+    disabled in order to use Spark local directories that reside on NFS filesystems (see
+    <a href="https://issues.apache.org/jira/browse/SPARK-6313">SPARK-6313</a>
for more details).
+  </td>
+</tr>
+<tr>
   <td><code>spark.files.overwrite</code></td>
   <td>false</td>
   <td>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message