spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Juliet Hougland (JIRA)" <>
Subject [jira] [Commented] (SPARK-8646) PySpark does not run on YARN
Date Thu, 16 Jul 2015 00:03:05 GMT


Juliet Hougland commented on SPARK-8646:

Yea, it works fine if I add that arg. There are two reasons I think this should be fixed in
Spark, despite there being a work around. First, I think API compatibility should include
scripts that 

> PySpark does not run on YARN
> ----------------------------
>                 Key: SPARK-8646
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, YARN
>    Affects Versions: 1.4.0
>         Environment: SPARK_HOME=local/path/to/spark1.4install/dir
> also with
> SPARK_HOME=local/path/to/spark1.4install/dir
> Spark apps are submitted with the command:
> $SPARK_HOME/bin/spark-submit outofstock/ hdfs://foe-dev/DEMO_DATA/FACT_POS
hdfs:/user/juliet/ex/ yarn-client
> data_transform contains a main method, and the rest of the args are parsed in my own
>            Reporter: Juliet Hougland
>         Attachments: executor.log, pi-test.log, spark1.4-SPARK_HOME-set-PYTHONPATH-set.log,
spark1.4-SPARK_HOME-set-inline-HADOOP_CONF_DIR.log, spark1.4-SPARK_HOME-set.log, spark1.4-verbose.log,
> Running pyspark jobs result in a "no module named pyspark" when run in yarn-client mode
in spark 1.4.
> [I believe this JIRA represents the change that introduced this error.|
> This does not represent a binary compatible change to spark. Scripts that worked on previous
spark versions (ie comands the use spark-submit) should continue to work without modification
between minor versions.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message