spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sun Rui (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (SPARK-8041) Consistently pass SparkR library directory to SparkR application
Date Tue, 30 Jun 2015 07:36:05 GMT

     [ https://issues.apache.org/jira/browse/SPARK-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sun Rui closed SPARK-8041.
--------------------------
    Resolution: Duplicate

This issue is covered by SPARK-6797

> Consistently pass SparkR library directory to SparkR application
> ----------------------------------------------------------------
>
>                 Key: SPARK-8041
>                 URL: https://issues.apache.org/jira/browse/SPARK-8041
>             Project: Spark
>          Issue Type: Improvement
>          Components: SparkR
>    Affects Versions: 1.4.0
>            Reporter: Sun Rui
>
> The SparkR package library directory path (RLibDir) is needed for SparkR applications
for loading SparkR package and locating R helper files inside the package.
> Currently, there are some places that the RLibDir needs to be specified.
> First of all, when you programs a SparkR application, sparkR.init() allows you to pass
a RLibDir parameter (by default, it is the same as the SparkR package's libname on the driver
host). However, it seems not reasonable to hard-code RLibDir in a program. Instead, it would
be more flexible to pass RLibDir via command line or env variable.
> Additionally, for YARN cluster mode, RRunner depends on SPARK_HOME env variable to get
the RLibDir (assume $SPARK_HOME/R/lib). 
> So it would be better to define a consistent way to pass RLibDir to a SparkR application
in all deployment modes. It could be a command line option for bin/sparkR or an env variable.
It can be passed to a sparkR application, and we can remove the RLibDir parameter of sparkR.init().
When in YARN cluster mode, it can be passed to AM using spark.yarn.appMasterEnv.[EnvironmentVariableName]
configuration option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message