spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shivaram Venkataraman (JIRA)" <>
Subject [jira] [Resolved] (SPARK-10500) cannot be created if $SPARK_HOME/R/lib is unwritable
Date Mon, 16 Nov 2015 03:30:10 GMT


Shivaram Venkataraman resolved SPARK-10500.
       Resolution: Fixed
    Fix Version/s: 1.6.0

Issue resolved by pull request 9390

> cannot be created if $SPARK_HOME/R/lib is unwritable
> ---------------------------------------------------------------
>                 Key: SPARK-10500
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SparkR
>    Affects Versions: 1.5.0
>            Reporter: Jonathan Kelly
>             Fix For: 1.6.0
> As of SPARK-6797, is re-created each time spark-submit is run with an R application,
which fails if Spark has been installed into a directory to which the current user doesn't
have write permissions. (e.g., on EMR's emr-4.0.0 release, Spark is installed at /usr/lib/spark,
which is only writable by root.)
> Would it be possible to skip creating if it already exists? That would enable to be pre-created by the root user and then reused each time spark-submit is run,
which I believe is similar to how pyspark works.
> Another option would be to make the location configurable, as it's currently hardcoded
to $SPARK_HOME/R/lib/ Allowing it to be configured to something like the user's
home directory or a random path in /tmp would get around the permissions issue.
> By the way, why does spark-submit even need to re-create every time a new
R application is launched? This seems unnecessary and inefficient, unless you are actively
developing the SparkR libraries and expect the contents of to change.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message