zeppelin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zjf...@apache.org
Subject zeppelin git commit: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui.SparkUI.appUIAddress() for the spark master
Date Sat, 08 Apr 2017 10:14:03 GMT
Repository: zeppelin
Updated Branches:
  refs/heads/branch-0.7 d0b4287d5 -> 46b196575


ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui.SparkUI.appUIAddress() for the spark
master

`SparkUI.addUIAddress` is removed in spark master which cause this error. Actually spark 2.0
introduce new api `SparkContext.uiWebUrl` (SPARK-14576) which is public in contrast `SparkUI.addUIAddress`
is private. This PR would use `SparkUI.addUIAddress` for spark 1 and `SparkContext.uiWebUrl`
 for spark 2.

[Improvement | Refactoring]

* [ ] - Task

* https://issues.apache.org/jira/browse/ZEPPELIN-2150

Manually verify it on spark 1.6.2, spark 2.1.0 and spark master

* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Jeff Zhang <zjffdu@apache.org>

Closes #2231 from zjffdu/ZEPPELIN-2150 and squashes the following commits:

499b623 [Jeff Zhang] return null when sparkUIUrl is not avaialbe
e285278 [Jeff Zhang] ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui.SparkUI.appUIAddress()
for the spark master

(cherry picked from commit 495be1ede1c0f5fc949275153dcaa1c6de5ddcdd)
Signed-off-by: Jeff Zhang <zjffdu@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/46b19657
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/46b19657
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/46b19657

Branch: refs/heads/branch-0.7
Commit: 46b1965752522f255e34002dbd9288cca1495b44
Parents: d0b4287
Author: Jeff Zhang <zjffdu@apache.org>
Authored: Sat Apr 8 08:15:14 2017 +0800
Committer: Jeff Zhang <zjffdu@apache.org>
Committed: Sat Apr 8 18:12:33 2017 +0800

----------------------------------------------------------------------
 .../apache/zeppelin/spark/SparkInterpreter.java | 22 +++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/zeppelin/blob/46b19657/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
----------------------------------------------------------------------
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
index 47f8080..a2eeac6 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
@@ -961,11 +961,23 @@ public class SparkInterpreter extends Interpreter {
     numReferenceOfSparkContext.incrementAndGet();
   }
 
-  private String getSparkUIUrl() {
-    Option<SparkUI> sparkUiOption = (Option<SparkUI>) Utils.invokeMethod(sc,
"ui");
-    SparkUI sparkUi = sparkUiOption.get();
-    String sparkWebUrl = sparkUi.appUIAddress();
-    return sparkWebUrl;
+  public String getSparkUIUrl() {
+    if (sparkUrl != null) {
+      return sparkUrl;
+    }
+
+    if (sparkVersion.newerThanEquals(SparkVersion.SPARK_2_0_0)) {
+      Option<String> uiWebUrlOption = (Option<String>) Utils.invokeMethod(sc,
"uiWebUrl");
+      if (uiWebUrlOption.isDefined()) {
+        return uiWebUrlOption.get();
+      }
+    } else {
+      Option<SparkUI> sparkUIOption = (Option<SparkUI>) Utils.invokeMethod(sc,
"ui");
+      if (sparkUIOption.isDefined()) {
+        return (String) Utils.invokeMethod(sparkUIOption.get(), "appUIAddress");
+      }
+    }
+    return null;
   }
 
   private Results.Result interpret(String line) {


Mime
View raw message