spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tgra...@apache.org
Subject git commit: [SPARK-3979] [yarn] Use fs's default replication.
Date Fri, 17 Oct 2014 18:45:21 GMT
Repository: spark
Updated Branches:
  refs/heads/master c35186206 -> 803e7f087


[SPARK-3979] [yarn] Use fs's default replication.

This avoids issues when HDFS is configured in a way that would not
allow the hardcoded default replication of "3".

Note: getDefaultReplication(Path) was added in 0.23.3, and the oldest
one available on Maven Central is 0.23.7, so I chose to not add code
to access that method via reflection.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #2831 from vanzin/SPARK-3979 and squashes the following commits:

b0e3a97 [Marcelo Vanzin] [SPARK-3979] [yarn] Use fs's default replication.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/803e7f08
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/803e7f08
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/803e7f08

Branch: refs/heads/master
Commit: 803e7f087797bae643754f8db88848a17282ca6e
Parents: c351862
Author: Marcelo Vanzin <vanzin@cloudera.com>
Authored: Fri Oct 17 13:45:10 2014 -0500
Committer: Thomas Graves <tgraves@apache.org>
Committed: Fri Oct 17 13:45:10 2014 -0500

----------------------------------------------------------------------
 .../src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/803e7f08/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
----------------------------------------------------------------------
diff --git a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
index 14a0386..0efac4e 100644
--- a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
+++ b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
@@ -143,7 +143,8 @@ private[spark] trait ClientBase extends Logging {
     val nns = getNameNodesToAccess(sparkConf) + dst
     obtainTokensForNamenodes(nns, hadoopConf, credentials)
 
-    val replication = sparkConf.getInt("spark.yarn.submit.file.replication", 3).toShort
+    val replication = sparkConf.getInt("spark.yarn.submit.file.replication",
+      fs.getDefaultReplication(dst)).toShort
     val localResources = HashMap[String, LocalResource]()
     FileSystem.mkdirs(fs, dst, new FsPermission(STAGING_DIR_PERMISSION))
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message