spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject git commit: [SPARK-1089] fix the regression problem on ADD_JARS in 0.9
Date Thu, 27 Feb 2014 07:42:47 GMT
Repository: spark
Updated Branches:
  refs/heads/master 6ccd6c55b -> 345df5f4a

[SPARK-1089] fix the regression problem on ADD_JARS in 0.9

copied from JIRA, reported by @ash211

"Using the ADD_JARS environment variable with spark-shell used to add the jar to both the
shell and the various workers. Now it only adds to the workers and importing a custom class
in the shell is broken.
The workaround is to add custom jars to both ADD_JARS and SPARK_CLASSPATH.
We should fix ADD_JARS so it works properly again.
See various threads on the user list:
(another one that doesn't appear in the archives yet titled "ADD_JARS not working on 0.9")"

The reason of this bug is two-folds

in the current implementation of SparkILoop.scala, the settings.classpath is not set properly
when the process() method is invoked

the weird behaviour of Scala 2.10, (I personally thought it is a bug)

if we simply set value of a PathSettings object (like settings.classpath), the isDefault is
not set to true (this is a flag showing if the variable is modified), so it makes the PathResolver
loads the default CLASSPATH environment variable value to calculated the path (see

what we have to do is to manually make this flag set, (

Author: CodingCat <>

Closes #13 from CodingCat/SPARK-1089 and squashes the following commits:

8af81e7 [CodingCat] impose non-null settings
9aa2125 [CodingCat] code cleaning
ce36676 [CodingCat] code cleaning
e045582 [CodingCat] fix the regression problem on ADD_JARS in 0.9


Branch: refs/heads/master
Commit: 345df5f4a9c16a6a87440afa2b09082fc3d224bd
Parents: 6ccd6c5
Author: CodingCat <>
Authored: Wed Feb 26 23:42:15 2014 -0800
Committer: Patrick Wendell <>
Committed: Wed Feb 26 23:42:15 2014 -0800

 repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
index 013cea0..f52ebe4 100644
--- a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
+++ b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
@@ -182,8 +182,13 @@ class SparkILoop(in0: Option[BufferedReader], protected val out: JPrintWriter,
   /** Create a new interpreter. */
   def createInterpreter() {
-    if (addedClasspath != "")
-      settings.classpath append addedClasspath
+    require(settings != null)
+    if (addedClasspath != "") settings.classpath.append(addedClasspath)
+    // work around for Scala bug
+    val totalClassPath = SparkILoop.getAddedJars.foldLeft(
+      settings.classpath.value)((l, r) => ClassPath.join(l, r))
+    this.settings.classpath.value = totalClassPath
     intp = new SparkILoopInterpreter

View raw message