spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject git commit: [SPARK-1134] Fix and document passing of arguments to IPython
Date Thu, 03 Apr 2014 22:49:00 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-0.9 28e764379 -> d9c7a808c

[SPARK-1134] Fix and document passing of arguments to IPython

This is based on @dianacarroll's previous pull request,
and @joshrosen's comments on Since we do want to
allow passing arguments to IPython, this does the following:
* It documents that IPython can't be used with standalone jobs for now. (Later versions of
IPython will deal with PYTHONSTARTUP properly and enable this, see,
but no released version has that fix.)
* If you run `pyspark` with `IPYTHON=1`, it passes your command-line arguments to it. This
way you can do stuff like `IPYTHON=1 bin/pyspark notebook`.
* The old `IPYTHON_OPTS` remains, but I've removed it from the documentation. This is in case
people read an old tutorial that uses it.

This is not a perfect solution and I'd also be okay with keeping things as they are today
(ignoring `$@` for IPython and using IPYTHON_OPTS), and only doing the doc change. With this
change though, when IPython fixes, people will
immediately be able to do `IPYTHON=1 bin/pyspark` to run a standalone script and
get all the benefits of running scripts in IPython (presumably better debugging and such).
Without it, there will be no way to run scripts in IPython.

@joshrosen you should probably take the final call on this.

Author: Diana Carroll <>

Closes #294 from mateiz/spark-1134 and squashes the following commits:

747bb13 [Diana Carroll] SPARK-1134 bug with ipython prevents non-interactive use with spark;
only call ipython if no command line arguments were supplied

(cherry picked from commit a599e43d6e0950f6b6b32150ce264a8c2711470c)
Signed-off-by: Matei Zaharia <>


Branch: refs/heads/branch-0.9
Commit: d9c7a808c0c6c329f6898b8b50f8cb7a35b17030
Parents: 28e7643
Author: Diana Carroll <>
Authored: Thu Apr 3 15:48:42 2014 -0700
Committer: Matei Zaharia <>
Committed: Thu Apr 3 15:48:51 2014 -0700

 bin/pyspark | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/bin/pyspark b/bin/pyspark
index ed6f8da..5a12516 100755
--- a/bin/pyspark
+++ b/bin/pyspark
@@ -58,7 +58,8 @@ if [ -n "$IPYTHON_OPTS" ]; then
-if [[ "$IPYTHON" = "1" ]] ; then
+# Only use ipython if no command line arguments were provided [SPARK-1134]
+if [[ "$IPYTHON" = "1" && $# = 0 ]] ; then
   exec ipython $IPYTHON_OPTS
   exec "$PYSPARK_PYTHON" "$@"

View raw message