spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <>
Subject [jira] [Created] (SPARK-1900) Fix running PySpark files on YARN
Date Thu, 22 May 2014 08:32:38 GMT
Andrew Or created SPARK-1900:

             Summary: Fix running PySpark files on YARN 
                 Key: SPARK-1900
             Project: Spark
          Issue Type: Bug
            Reporter: Andrew Or
            Priority: Blocker

This fails currently because of a mismatch in paths.

On a YARN cluster, spark-submit automatically assumes the file is on HDFS, even if it is a
relative path that refers to a local file. A natural workaround for this is to explicitly
specify the "file:" prefix. However, this prefix is not understood by python, which fails
with the following:

python: can't open file 'file:path/to/my/': [Errno 2] No such file or directory

This message was sent by Atlassian JIRA

View raw message