spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From joshro...@apache.org
Subject spark git commit: [SPARK-4764] Ensure that files are fetched atomically
Date Mon, 08 Dec 2014 19:50:41 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.1 8ee2d1873 -> 16bc77b7d


[SPARK-4764] Ensure that files are fetched atomically

tempFile is created in the same directory than targetFile, so that the
move from tempFile to targetFile is always atomic

Author: Christophe Préaud <christophe.preaud@kelkoo.com>

Closes #2855 from preaudc/master and squashes the following commits:

9ba89ca [Christophe Préaud] Ensure that files are fetched atomically
54419ae [Christophe Préaud] Merge remote-tracking branch 'upstream/master'
c6a5590 [Christophe Préaud] Revert commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd
7456a33 [Christophe Préaud] Merge remote-tracking branch 'upstream/master'
8ea871f [Christophe Préaud] Ensure that files are fetched atomically

(cherry picked from commit ab2abcb5ef925f15fa0e08d34a79b94a7b6578ef)
Signed-off-by: Josh Rosen <rosenville@gmail.com>

Conflicts:
	core/src/main/scala/org/apache/spark/util/Utils.scala


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/16bc77b7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/16bc77b7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/16bc77b7

Branch: refs/heads/branch-1.1
Commit: 16bc77b7d91d9b21b9be45d346c47443386295d9
Parents: 8ee2d18
Author: Christophe Préaud <christophe.preaud@kelkoo.com>
Authored: Mon Dec 8 11:44:54 2014 -0800
Committer: Josh Rosen <rosenville@gmail.com>
Committed: Mon Dec 8 11:50:10 2014 -0800

----------------------------------------------------------------------
 core/src/main/scala/org/apache/spark/util/Utils.scala | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/16bc77b7/core/src/main/scala/org/apache/spark/util/Utils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala
index ef11ed3..20bf36e 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -378,8 +378,7 @@ private[spark] object Utils extends Logging {
    */
   def fetchFile(url: String, targetDir: File, conf: SparkConf, securityMgr: SecurityManager)
{
     val filename = url.split("/").last
-    val tempDir = getLocalDir(conf)
-    val tempFile =  File.createTempFile("fetchFileTemp", null, new File(tempDir))
+    val tempFile = File.createTempFile("fetchFileTemp", null, new File(targetDir.getAbsolutePath))
     val targetFile = new File(targetDir, filename)
     val uri = new URI(url)
     val fileOverwrite = conf.getBoolean("spark.files.overwrite", false)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message