zeppelin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From b..@apache.org
Subject [2/3] incubator-zeppelin git commit: [ZEPPELIN-572] pyspark interpreter doesn't work on yarn-client
Date Mon, 11 Jan 2016 13:22:11 GMT
[ZEPPELIN-572] pyspark interpreter doesn't work on yarn-client

### What is this PR for?
Set `spark.yarn.isPython` to be `true` to distribute pyspark libraries to workers when master
is `yarn-client`

### What type of PR is it?
Bug Fix

### Is there a relevant Jira issue?
[ZEPPELIN-572](https://issues.apache.org/jira/browse/ZEPPELIN-572)

### How should this be tested?
You need yarn cluster to test this PR.
Simple way to test this PR would be running below code in paragraph and see if it throws error.
```
%pyspark
print(sc.parallelize([1, 2]).count())
```
And you should be able to see that `spark.yarn.isPython` is set to `true` in **Spark UI >
Environment > Spark Properties** only when you set spark.master as `yarn-client`.

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Mina Lee <minalee@nflabs.com>

Closes #605 from minahlee/ZEPPELIN-572 and squashes the following commits:

8c99de5 [Mina Lee] Set spark.yarn.isPython to be true to distribute needed pyspark libraries
to workers when master is yarn-client


Project: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/commit/ed1cc023
Tree: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/tree/ed1cc023
Diff: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/diff/ed1cc023

Branch: refs/heads/branch-0.5.6
Commit: ed1cc023953834ff29d0bc212d83f0fe8d000eff
Parents: dedd6a6
Author: Mina Lee <minalee@nflabs.com>
Authored: Wed Jan 6 21:23:18 2016 -0800
Committer: Alexander Bezzubov <bzz@apache.org>
Committed: Mon Jan 11 22:13:14 2016 +0900

----------------------------------------------------------------------
 .../main/java/org/apache/zeppelin/spark/SparkInterpreter.java    | 4 ++++
 1 file changed, 4 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/ed1cc023/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
----------------------------------------------------------------------
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
index 2f2829e..b967c29 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
@@ -335,6 +335,10 @@ public class SparkInterpreter extends Interpreter {
       conf.set("spark.submit.pyArchives", Joiner.on(":").join(pythonLibs));
     }
 
+    // Distributes needed libraries to workers.
+    if (getProperty("master").equals("yarn-client")) {
+      conf.set("spark.yarn.isPython", "true");
+    }
 
     SparkContext sparkContext = new SparkContext(conf);
     return sparkContext;


Mime
View raw message