zeppelin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zjf...@apache.org
Subject zeppelin git commit: ZEPPELIN-3633. ZeppelinContext Not Found in yarn-cluster Mode
Date Wed, 26 Sep 2018 00:52:43 GMT
Repository: zeppelin
Updated Branches:
  refs/heads/master 68512b9cd -> 92f244ef7


ZEPPELIN-3633. ZeppelinContext Not Found in yarn-cluster Mode

### What is this PR for?
This issue is due to classpath in cluster mode. Because in cluster mode, driver run in the
node of yarn cluster which don't have zeppelin installed. This PR fix this issue by updating
the classpath of spark repl.

### What type of PR is it?
[Bug Fix ]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3633

### How should this be tested?
* Manually tested. Unfortunately this only happens on multiple node cluster, no unit test
can be added for this scenario.

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Jeff Zhang <zjffdu@apache.org>

Closes #3181 from zjffdu/ZEPPELIN-3633 and squashes the following commits:

e800037d9 [Jeff Zhang] ZEPPELIN-3633. ZeppelinContext Not Found in yarn-cluster Mode


Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/92f244ef
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/92f244ef
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/92f244ef

Branch: refs/heads/master
Commit: 92f244ef7e1902e51dbec6b759152341992d834c
Parents: 68512b9
Author: Jeff Zhang <zjffdu@apache.org>
Authored: Wed Sep 19 11:21:50 2018 +0800
Committer: Jeff Zhang <zjffdu@apache.org>
Committed: Wed Sep 26 08:52:36 2018 +0800

----------------------------------------------------------------------
 spark/interpreter/figure/unnamed-chunk-1-1.png    | Bin 0 -> 407541 bytes
 .../spark/BaseSparkScalaInterpreter.scala         |  10 +++++++++-
 2 files changed, 9 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/zeppelin/blob/92f244ef/spark/interpreter/figure/unnamed-chunk-1-1.png
----------------------------------------------------------------------
diff --git a/spark/interpreter/figure/unnamed-chunk-1-1.png b/spark/interpreter/figure/unnamed-chunk-1-1.png
new file mode 100644
index 0000000..6f03c95
Binary files /dev/null and b/spark/interpreter/figure/unnamed-chunk-1-1.png differ

http://git-wip-us.apache.org/repos/asf/zeppelin/blob/92f244ef/spark/spark-scala-parent/src/main/scala/org/apache/zeppelin/spark/BaseSparkScalaInterpreter.scala
----------------------------------------------------------------------
diff --git a/spark/spark-scala-parent/src/main/scala/org/apache/zeppelin/spark/BaseSparkScalaInterpreter.scala
b/spark/spark-scala-parent/src/main/scala/org/apache/zeppelin/spark/BaseSparkScalaInterpreter.scala
index a73630a..2cbda93 100644
--- a/spark/spark-scala-parent/src/main/scala/org/apache/zeppelin/spark/BaseSparkScalaInterpreter.scala
+++ b/spark/spark-scala-parent/src/main/scala/org/apache/zeppelin/spark/BaseSparkScalaInterpreter.scala
@@ -372,8 +372,16 @@ abstract class BaseSparkScalaInterpreter(val conf: SparkConf,
     val sparkJars = conf.getOption("spark.jars").map(_.split(","))
       .map(_.filter(_.nonEmpty)).toSeq.flatten
     val depJars = depFiles.asScala.filter(_.endsWith(".jar"))
-    val result = sparkJars ++ depJars
+    // add zeppelin spark interpreter jar
+    val zeppelinInterpreterJarURL = getClass.getProtectionDomain.getCodeSource.getLocation
+    // zeppelinInterpreterJarURL might be a folder when under unit testing
+    val result = if (new File(zeppelinInterpreterJarURL.getFile).isDirectory) {
+      sparkJars ++ depJars
+    } else {
+      sparkJars ++ depJars ++ Seq(zeppelinInterpreterJarURL.getFile)
+    }
     conf.set("spark.jars", result.mkString(","))
+    LOGGER.debug("User jar for spark repl: " + conf.get("spark.jars"))
     result
   }
 


Mime
View raw message