zeppelin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From felixche...@apache.org
Subject zeppelin git commit: [ZEPPELIN-2949] Allow custom Spark UI URL
Date Tue, 24 Oct 2017 06:25:12 GMT
Repository: zeppelin
Updated Branches:
  refs/heads/master 84cb4b5fb -> c45128e79


[ZEPPELIN-2949] Allow custom Spark UI URL

### What is this PR for?
Allow an override of default Spark logic in building the URL path and redirect the Zeppelin
user directly to a configured reverse proxy path

### What type of PR is it?
[Improvement]

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2949

### How should this be tested?
* Set "zeppelin.spark.uiWebUrl" to http://localhost/spark-reverse-proxy-path
* Run Spark code
* Open Spark UI
* Expected to be redirected to http://localhost/spark-reverse-proxy-path

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? N
* Is there breaking changes for older versions? N
* Does this needs documentation? N

Author: Nelson Costa <nelson.costa85@gmail.com>

Closes #2596 from necosta/zeppelin2949 and squashes the following commits:

2ae8f2d [Nelson Costa] [ZEPPELIN-2949] Last amendments
118339f [Nelson Costa] [ZEPPELIN-2949] Updated documentation
826bad4 [Nelson Costa] [ZEPPELIN-2949] Updated documentation
0a56299 [Nelson Costa] [ZEPPELIN-2949] Allow custom Spark UI URL


Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/c45128e7
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/c45128e7
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/c45128e7

Branch: refs/heads/master
Commit: c45128e79f619e6f2d927176a440ec44a2d588ed
Parents: 84cb4b5
Author: Nelson Costa <nelson.costa85@gmail.com>
Authored: Thu Oct 19 10:21:54 2017 +0100
Committer: Felix Cheung <felixcheung@apache.org>
Committed: Mon Oct 23 23:24:55 2017 -0700

----------------------------------------------------------------------
 docs/interpreter/spark.md                       | 21 ++++++++++++--------
 .../apache/zeppelin/spark/SparkInterpreter.java | 11 +++++-----
 .../src/main/resources/interpreter-setting.json |  7 +++++++
 .../zeppelin/spark/SparkInterpreterTest.java    |  2 +-
 4 files changed, 27 insertions(+), 14 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c45128e7/docs/interpreter/spark.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/spark.md b/docs/interpreter/spark.md
index bbd9065..da957c6 100644
--- a/docs/interpreter/spark.md
+++ b/docs/interpreter/spark.md
@@ -145,6 +145,11 @@ You can also set other Spark properties which are not listed in the table.
For a
     <td>true</td>
     <td>Do not change - developer only setting, not for production use</td>
   </tr>
+  <tr>
+  <td>zeppelin.spark.uiWebUrl</td>
+    <td></td>
+    <td>Overrides Spark UI default URL. Value should be a full URL (ex: http://{hostName}/{uniquePath}</td>
+  </tr>
 </table>
 
 Without any configuration, Spark interpreter works out of box in local mode. But if you want
to connect to your Spark cluster, you'll need to follow below two simple steps.
@@ -184,7 +189,7 @@ For example,
  * **yarn-cluster** in Yarn cluster mode
  * **mesos://host:5050** in Mesos cluster
 
-That's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding
Zeppelin in this way. 
+That's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding
Zeppelin in this way.
 For the further information about Spark & Zeppelin version compatibility, please refer
to "Available Interpreters" section in [Zeppelin download page](https://zeppelin.apache.org/download.html).
 
 > Note that without exporting `SPARK_HOME`, it's running in local mode with included version
of Spark. The included version may vary depending on the build profile.
@@ -216,7 +221,7 @@ There are two ways to load external libraries in Spark interpreter. First
is usi
 Please see [Dependency Management](../usage/interpreter/dependency_management.html) for the
details.
 
 ### 2. Loading Spark Properties
-Once `SPARK_HOME` is set in `conf/zeppelin-env.sh`, Zeppelin uses `spark-submit` as spark
interpreter runner. `spark-submit` supports two ways to load configurations. 
+Once `SPARK_HOME` is set in `conf/zeppelin-env.sh`, Zeppelin uses `spark-submit` as spark
interpreter runner. `spark-submit` supports two ways to load configurations.
 The first is command line options such as --master and Zeppelin can pass these options to
`spark-submit` by exporting `SPARK_SUBMIT_OPTIONS` in `conf/zeppelin-env.sh`. Second is reading
configuration options from `SPARK_HOME/conf/spark-defaults.conf`. Spark properties that user
can set to distribute libraries are:
 
 <table class="table-configuration">
@@ -249,7 +254,7 @@ Here are few examples:
   ```bash
     export SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0 --jars /path/mylib1.jar,/path/mylib2.jar
--files /path/mylib1.py,/path/mylib2.zip,/path/mylib3.egg"
   ```
-    
+
 * `SPARK_HOME/conf/spark-defaults.conf`
 
   ```
@@ -414,17 +419,17 @@ To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/
 
 
 ## Matplotlib Integration (pyspark)
-Both the `python` and `pyspark` interpreters have built-in support for inline visualization
using `matplotlib`, 
-a popular plotting library for python. More details can be found in the [python interpreter
documentation](../interpreter/python.html), 
-since matplotlib support is identical. More advanced interactive plotting can be done with
pyspark through 
+Both the `python` and `pyspark` interpreters have built-in support for inline visualization
using `matplotlib`,
+a popular plotting library for python. More details can be found in the [python interpreter
documentation](../interpreter/python.html),
+since matplotlib support is identical. More advanced interactive plotting can be done with
pyspark through
 utilizing Zeppelin's built-in [Angular Display System](../usage/display_system/angular_backend.html),
as shown below:
 
 <img class="img-responsive" src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/matplotlibAngularExample.gif"
/>
 
 ## Interpreter setting option
 
-You can choose one of `shared`, `scoped` and `isolated` options wheh you configure Spark
interpreter. 
-Spark interpreter creates separated Scala compiler per each notebook but share a single SparkContext
in `scoped` mode (experimental). 
+You can choose one of `shared`, `scoped` and `isolated` options wheh you configure Spark
interpreter.
+Spark interpreter creates separated Scala compiler per each notebook but share a single SparkContext
in `scoped` mode (experimental).
 It creates separated SparkContext per each notebook in `isolated` mode.
 
 ## IPython support

http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c45128e7/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
----------------------------------------------------------------------
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
index 5a5110f..0536825 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
@@ -26,11 +26,9 @@ import java.lang.reflect.InvocationTargetException;
 import java.lang.reflect.Method;
 import java.net.URL;
 import java.net.URLClassLoader;
-import java.util.ArrayList;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
-import java.util.NoSuchElementException;
 import java.util.Properties;
 import java.util.Set;
 import java.util.concurrent.atomic.AtomicInteger;
@@ -51,7 +49,6 @@ import org.apache.spark.sql.SQLContext;
 import org.apache.spark.ui.SparkUI;
 import org.apache.spark.ui.jobs.JobProgressListener;
 import org.apache.zeppelin.interpreter.BaseZeppelinContext;
-import org.apache.zeppelin.interpreter.DefaultInterpreterProperty;
 import org.apache.zeppelin.interpreter.Interpreter;
 import org.apache.zeppelin.interpreter.InterpreterContext;
 import org.apache.zeppelin.interpreter.InterpreterException;
@@ -72,7 +69,6 @@ import org.apache.zeppelin.spark.dep.SparkDependencyResolver;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import com.google.common.base.Joiner;
 import scala.Console;
 import scala.Enumeration.Value;
 import scala.None;
@@ -206,7 +202,7 @@ public class SparkInterpreter extends Interpreter {
       private String getJobUrl(int jobId) {
         String jobUrl = null;
         if (sparkUrl != null) {
-          jobUrl = sparkUrl + "/jobs/job?id=" + jobId;
+          jobUrl = sparkUrl + "/jobs/job/?id=" + jobId;
         }
         return jobUrl;
       }
@@ -936,6 +932,11 @@ public class SparkInterpreter extends Interpreter {
       return sparkUrl;
     }
 
+    String sparkUrlProp = property.getProperty("zeppelin.spark.uiWebUrl", "");
+    if (!StringUtils.isBlank(sparkUrlProp)) {
+      return sparkUrlProp;
+    }
+
     if (sparkVersion.newerThanEquals(SparkVersion.SPARK_2_0_0)) {
       Option<String> uiWebUrlOption = (Option<String>) Utils.invokeMethod(sc,
"uiWebUrl");
       if (uiWebUrlOption.isDefined()) {

http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c45128e7/spark/src/main/resources/interpreter-setting.json
----------------------------------------------------------------------
diff --git a/spark/src/main/resources/interpreter-setting.json b/spark/src/main/resources/interpreter-setting.json
index 4aeda6c..485f695 100644
--- a/spark/src/main/resources/interpreter-setting.json
+++ b/spark/src/main/resources/interpreter-setting.json
@@ -67,6 +67,13 @@
         "defaultValue": true,
         "description": "Do not change - developer only setting, not for production use",
         "type": "checkbox"
+      },
+      "zeppelin.spark.uiWebUrl": {
+        "envName": null,
+        "propertyName": "zeppelin.spark.uiWebUrl",
+        "defaultValue": "",
+        "description": "Override Spark UI default URL",
+        "type": "string"
       }
     },
     "editor": {

http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c45128e7/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
----------------------------------------------------------------------
diff --git a/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java b/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
index aaf13c3..10141a1 100644
--- a/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
+++ b/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
@@ -347,7 +347,7 @@ public class SparkInterpreterTest {
     }
     String sparkUIUrl = repl.getSparkUIUrl();
     assertNotNull(jobUrl);
-    assertTrue(jobUrl.startsWith(sparkUIUrl + "/jobs/job?id="));
+    assertTrue(jobUrl.startsWith(sparkUIUrl + "/jobs/job/?id="));
 
   }
 }


Mime
View raw message