spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ma...@apache.org
Subject git commit: [SPARK-1134] Fix and document passing of arguments to IPython
Date Thu, 03 Apr 2014 22:49:00 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-0.9 28e764379 -> d9c7a808c


[SPARK-1134] Fix and document passing of arguments to IPython

This is based on @dianacarroll's previous pull request https://github.com/apache/spark/pull/227,
and @joshrosen's comments on https://github.com/apache/spark/pull/38. Since we do want to
allow passing arguments to IPython, this does the following:
* It documents that IPython can't be used with standalone jobs for now. (Later versions of
IPython will deal with PYTHONSTARTUP properly and enable this, see https://github.com/ipython/ipython/pull/5226,
but no released version has that fix.)
* If you run `pyspark` with `IPYTHON=1`, it passes your command-line arguments to it. This
way you can do stuff like `IPYTHON=1 bin/pyspark notebook`.
* The old `IPYTHON_OPTS` remains, but I've removed it from the documentation. This is in case
people read an old tutorial that uses it.

This is not a perfect solution and I'd also be okay with keeping things as they are today
(ignoring `$@` for IPython and using IPYTHON_OPTS), and only doing the doc change. With this
change though, when IPython fixes https://github.com/ipython/ipython/pull/5226, people will
immediately be able to do `IPYTHON=1 bin/pyspark myscript.py` to run a standalone script and
get all the benefits of running scripts in IPython (presumably better debugging and such).
Without it, there will be no way to run scripts in IPython.

@joshrosen you should probably take the final call on this.

Author: Diana Carroll <dcarroll@cloudera.com>

Closes #294 from mateiz/spark-1134 and squashes the following commits:

747bb13 [Diana Carroll] SPARK-1134 bug with ipython prevents non-interactive use with spark;
only call ipython if no command line arguments were supplied

(cherry picked from commit a599e43d6e0950f6b6b32150ce264a8c2711470c)
Signed-off-by: Matei Zaharia <matei@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d9c7a808
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d9c7a808
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d9c7a808

Branch: refs/heads/branch-0.9
Commit: d9c7a808c0c6c329f6898b8b50f8cb7a35b17030
Parents: 28e7643
Author: Diana Carroll <dcarroll@cloudera.com>
Authored: Thu Apr 3 15:48:42 2014 -0700
Committer: Matei Zaharia <matei@databricks.com>
Committed: Thu Apr 3 15:48:51 2014 -0700

----------------------------------------------------------------------
 bin/pyspark | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/d9c7a808/bin/pyspark
----------------------------------------------------------------------
diff --git a/bin/pyspark b/bin/pyspark
index ed6f8da..5a12516 100755
--- a/bin/pyspark
+++ b/bin/pyspark
@@ -58,7 +58,8 @@ if [ -n "$IPYTHON_OPTS" ]; then
   IPYTHON=1
 fi
 
-if [[ "$IPYTHON" = "1" ]] ; then
+# Only use ipython if no command line arguments were provided [SPARK-1134]
+if [[ "$IPYTHON" = "1" && $# = 0 ]] ; then
   exec ipython $IPYTHON_OPTS
 else
   exec "$PYSPARK_PYTHON" "$@"


Mime
View raw message