spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <>
Subject [jira] [Created] (SPARK-2111) pyspark errors when SPARK_PRINT_LAUNCH_COMMAND=1
Date Wed, 11 Jun 2014 16:04:02 GMT
Thomas Graves created SPARK-2111:

             Summary: pyspark errors when SPARK_PRINT_LAUNCH_COMMAND=1
                 Key: SPARK-2111
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 1.0.0
            Reporter: Thomas Graves

If you set SPARK_PRINT_LAUNCH_COMMAND=1 to see what java command is being used to launch spark
and then try to run pyspark it errors out with a very non-useful error message:

Traceback (most recent call last):
  File "/homes/tgraves/test/hadoop2/y-spark-git/python/pyspark/", line 43, in <module>
    sc = SparkContext(appName="PySparkShell", pyFiles=add_files)
  File "/homes/tgraves/test/hadoop2/y-spark-git/python/pyspark/", line 94, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway)
  File "/homes/tgraves/test/hadoop2/y-spark-git/python/pyspark/", line 184, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway()
  File "/homes/tgraves/test/hadoop2/y-spark-git/python/pyspark/", line 51,
in launch_gateway
    gateway_port = int(proc.stdout.readline())
ValueError: invalid literal for int() with base 10: 'Spark Command: /home/gs/java/jdk/bin/java
-cp :/home/gs/hadoop/current/share/hadoop/common/hadoop-gpl-compression.jar:/home/gs/hadoop/current/share/hadoop/hdfs/lib/YahooDNSToSwitchMapping-0.2.14020207'

This message was sent by Atlassian JIRA

View raw message