zeppelin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From m...@apache.org
Subject zeppelin git commit: ZEPPELIN-1342. Adding dependencies via SPARK_SUBMIT_OPTIONS doesn't work on Spark 2.0.0
Date Mon, 29 Aug 2016 00:27:17 GMT
Repository: zeppelin
Updated Branches:
  refs/heads/master b7f918a7b -> c1999ea80


ZEPPELIN-1342. Adding dependencies via SPARK_SUBMIT_OPTIONS doesn't work on Spark 2.0.0

### What is this PR for?

The root cause is due to the change of repl of scala-2.11. User needs to specify the jars
in the repl setting explicitly.

### What type of PR is it?
[Bug Fix]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1342

### How should this be tested?
Tested manually as shown in the screenshot.

### Screenshots (if appropriate)

![image](https://cloud.githubusercontent.com/assets/164491/17997416/f26c262c-6ba0-11e6-8586-22a6a633b21b.png)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Jeff Zhang <zjffdu@apache.org>

Closes #1366 from zjffdu/ZEPPELIN-1342 and squashes the following commits:

10f64e6 [Jeff Zhang] fix test failure
56925d6 [Jeff Zhang] ZEPPELIN-1342. Adding dependencies via SPARK_SUBMIT_OPTIONS doesn't work
on Spark 2.0.0


Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/c1999ea8
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/c1999ea8
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/c1999ea8

Branch: refs/heads/master
Commit: c1999ea806fdb2a0739dc566420253af5cab8c46
Parents: b7f918a
Author: Jeff Zhang <zjffdu@apache.org>
Authored: Sat Aug 27 10:34:50 2016 +0800
Committer: Lee moon soo <moon@apache.org>
Committed: Mon Aug 29 10:27:09 2016 +1000

----------------------------------------------------------------------
 .../main/java/org/apache/zeppelin/spark/SparkInterpreter.java | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c1999ea8/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
----------------------------------------------------------------------
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
index 6dc13ad..5320cae 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
@@ -33,6 +33,7 @@ import java.util.concurrent.atomic.AtomicInteger;
 
 import com.google.common.base.Joiner;
 
+import org.apache.commons.lang3.StringUtils;
 import org.apache.hadoop.security.UserGroupInformation;
 import org.apache.spark.SparkConf;
 import org.apache.spark.SparkContext;
@@ -593,7 +594,11 @@ public class SparkInterpreter extends Interpreter {
       argList.add("-Yrepl-class-based");
       argList.add("-Yrepl-outdir");
       argList.add(outputDir.getAbsolutePath());
-
+      if (conf.contains("spark.jars")) {
+        String jars = StringUtils.join(conf.get("spark.jars").split(","), File.separator);
+        argList.add("-classpath");
+        argList.add(jars);
+      }
 
       scala.collection.immutable.List<String> list =
           JavaConversions.asScalaBuffer(argList).toList();


Mime
View raw message