spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject git commit: [SPARK-3140] Clarify confusing PySpark exception message
Date Thu, 21 Aug 2014 00:07:49 GMT
Repository: spark
Updated Branches:
  refs/heads/master c9f743957 -> ba3c730e3


[SPARK-3140] Clarify confusing PySpark exception message

We read the py4j port from the stdout of the `bin/spark-submit` subprocess. If there is interference
in stdout (e.g. a random echo in `spark-submit`), we throw an exception with a warning message.
We do not, however, distinguish between this case from the case where no stdout is produced
at all.

I wasted a non-trivial amount of time being baffled by this exception in search of places
where I print random whitespace (in vain, of course). A clearer exception message that distinguishes
between these cases will prevent similar headaches that I have gone through.

Author: Andrew Or <andrewor14@gmail.com>

Closes #2067 from andrewor14/python-exception and squashes the following commits:

742f823 [Andrew Or] Further clarify warning messages
e96a7a0 [Andrew Or] Distinguish between unexpected output and no output at all


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ba3c730e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ba3c730e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ba3c730e

Branch: refs/heads/master
Commit: ba3c730e35bcdb662396955c3cc6f7de628034c8
Parents: c9f7439
Author: Andrew Or <andrewor14@gmail.com>
Authored: Wed Aug 20 17:07:39 2014 -0700
Committer: Andrew Or <andrewor14@gmail.com>
Committed: Wed Aug 20 17:07:39 2014 -0700

----------------------------------------------------------------------
 python/pyspark/java_gateway.py | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/ba3c730e/python/pyspark/java_gateway.py
----------------------------------------------------------------------
diff --git a/python/pyspark/java_gateway.py b/python/pyspark/java_gateway.py
index c7f7c1f..6f4f62f 100644
--- a/python/pyspark/java_gateway.py
+++ b/python/pyspark/java_gateway.py
@@ -54,12 +54,19 @@ def launch_gateway():
             gateway_port = proc.stdout.readline()
             gateway_port = int(gateway_port)
         except ValueError:
+            # Grab the remaining lines of stdout
             (stdout, _) = proc.communicate()
             exit_code = proc.poll()
             error_msg = "Launching GatewayServer failed"
-            error_msg += " with exit code %d! " % exit_code if exit_code else "! "
-            error_msg += "(Warning: unexpected output detected.)\n\n"
-            error_msg += gateway_port + stdout
+            error_msg += " with exit code %d!\n" % exit_code if exit_code else "!\n"
+            error_msg += "Warning: Expected GatewayServer to output a port, but found "
+            if gateway_port == "" and stdout == "":
+                error_msg += "no output.\n"
+            else:
+                error_msg += "the following:\n\n"
+                error_msg += "--------------------------------------------------------------\n"
+                error_msg += gateway_port + stdout
+                error_msg += "--------------------------------------------------------------\n"
             raise Exception(error_msg)
 
         # Create a thread to echo output from the GatewayServer, which is required


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message