spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From van...@apache.org
Subject spark git commit: [SPARK-20365][YARN] Remove local scheme when add path to ClassPath.
Date Thu, 01 Jun 2017 21:40:28 GMT
Repository: spark
Updated Branches:
  refs/heads/master f7cf2096f -> 640afa49a


[SPARK-20365][YARN] Remove local scheme when add path to ClassPath.

In Spark on YARN, when configuring "spark.yarn.jars" with local jars (jars started with "local"
scheme), we will get inaccurate classpath for AM and containers. This is because we don't
remove "local" scheme when concatenating classpath. It is OK to run because classpath is separated
with ":" and java treat "local" as a separate jar. But we could improve it to remove the scheme.

Updated `ClientSuite` to check "local" is not in the classpath.

cc jerryshao

Author: Li Yichao <lyc@zhihu.com>
Author: Li Yichao <liyichao.good@gmail.com>

Closes #18129 from liyichao/SPARK-20365.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/640afa49
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/640afa49
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/640afa49

Branch: refs/heads/master
Commit: 640afa49aa349c7ebe35d365eec3ef9bb7710b1d
Parents: f7cf209
Author: Li Yichao <lyc@zhihu.com>
Authored: Thu Jun 1 14:39:57 2017 -0700
Committer: Marcelo Vanzin <vanzin@cloudera.com>
Committed: Thu Jun 1 14:40:05 2017 -0700

----------------------------------------------------------------------
 .../yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 3 ++-
 .../src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/640afa49/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
----------------------------------------------------------------------
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
index 9956071..1fb7edf 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
@@ -1275,7 +1275,8 @@ private object Client extends Logging {
     if (sparkConf.get(SPARK_ARCHIVE).isEmpty) {
       sparkConf.get(SPARK_JARS).foreach { jars =>
         jars.filter(isLocalUri).foreach { jar =>
-          addClasspathEntry(getClusterPath(sparkConf, jar), env)
+          val uri = new URI(jar)
+          addClasspathEntry(getClusterPath(sparkConf, uri.getPath()), env)
         }
       }
     }

http://git-wip-us.apache.org/repos/asf/spark/blob/640afa49/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala
----------------------------------------------------------------------
diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala
b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala
index 3a11787..6cf6842 100644
--- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala
+++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala
@@ -122,6 +122,7 @@ class ClientSuite extends SparkFunSuite with Matchers with BeforeAndAfterAll
         cp should not contain (uri.getPath())
       }
     })
+    cp should not contain ("local")
     cp should contain(PWD)
     cp should contain (s"$PWD${Path.SEPARATOR}${LOCALIZED_CONF_DIR}")
     cp should not contain (APP_JAR)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message