spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From van...@apache.org
Subject spark git commit: [SPARK-21339][CORE] spark-shell --packages option does not add jars to classpath on windows
Date Tue, 01 Aug 2017 20:42:45 GMT
Repository: spark
Updated Branches:
  refs/heads/master b1d59e60d -> 58da1a245


[SPARK-21339][CORE] spark-shell --packages option does not add jars to classpath on windows

The --packages option jars are getting added to the classpath with the scheme as "file:///",
in Unix it doesn't have problem with this since the scheme contains the Unix Path separator
which separates the jar name with location in the classpath. In Windows, the jar file is not
getting resolved from the classpath because of the scheme.

Windows : file:///C:/Users/<user>/.ivy2/jars/<jar-name>.jar
Unix : file:///home/<user>/.ivy2/jars/<jar-name>.jar

With this PR, we are avoiding the 'file://' scheme to get added to the packages jar files.

I have verified manually in Windows and Unix environments, with the change it adds the jar
to classpath like below,

Windows : C:\Users\<user>\.ivy2\jars\<jar-name>.jar
Unix : /home/<user>/.ivy2/jars/<jar-name>.jar

Author: Devaraj K <devaraj@apache.org>

Closes #18708 from devaraj-kavali/SPARK-21339.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/58da1a24
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/58da1a24
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/58da1a24

Branch: refs/heads/master
Commit: 58da1a2455258156fe8ba57241611eac1a7928ef
Parents: b1d59e6
Author: Devaraj K <devaraj@apache.org>
Authored: Tue Aug 1 13:38:55 2017 -0700
Committer: Marcelo Vanzin <vanzin@cloudera.com>
Committed: Tue Aug 1 13:39:23 2017 -0700

----------------------------------------------------------------------
 .../scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/58da1a24/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
----------------------------------------------------------------------
diff --git a/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala b/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
index b8b38e8..9702a1e 100644
--- a/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
+++ b/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
@@ -18,6 +18,7 @@
 package org.apache.spark.repl
 
 import java.io.File
+import java.net.URI
 import java.util.Locale
 
 import scala.tools.nsc.GenericRunnerSettings
@@ -56,7 +57,10 @@ object Main extends Logging {
   // Visible for testing
   private[repl] def doMain(args: Array[String], _interp: SparkILoop): Unit = {
     interp = _interp
-    val jars = Utils.getUserJars(conf, isShell = true).mkString(File.pathSeparator)
+    val jars = Utils.getUserJars(conf, isShell = true)
+      // Remove file:///, file:// or file:/ scheme if exists for each jar
+      .map { x => if (x.startsWith("file:")) new File(new URI(x)).getPath else x }
+      .mkString(File.pathSeparator)
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir", s"${outputDir.getAbsolutePath}",


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message