spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jay vyas (JIRA)" <>
Subject [jira] [Commented] (SPARK-1807) Modify SPARK_EXECUTOR_URI to allow for script execution in Mesos.
Date Thu, 10 Jul 2014 14:26:04 GMT


jay vyas commented on SPARK-1807:

Question: Are you saying the {{SPARK_EXECUTOR_URI}} should be a URI that points to a script?
 Or that we can replace the URI with a simple path to an executable script.  

Proposed Alternative: Take  this extension to its logical extreme, and actually deprecate
the SPARK_EXECUTOR_URI parameter, and instead have a parameter {{SPARK_EXECUTOR_CLASS}}, which
is invoked with any number of args, one of which could be SPARK_EXECUTOR_URI.  

Something like this: 
val c = Class.forName(conf.get("SPARK_EXECUTOR_CLASS"))
val m = c.getMethod("startSpark",classOf[SparkExecutor])

The advantage of using a class, rather than a shell script, for this - is that we are gauranteed
off the bat that the class is providing a contract - to start spark - rather than just a one
off script which could do any number of things.  

Also it means that various classes for this can be maintained and tested over time as first
class "spark environment" providers.

> Modify SPARK_EXECUTOR_URI to allow for script execution in Mesos.
> -----------------------------------------------------------------
>                 Key: SPARK-1807
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Mesos
>    Affects Versions: 0.9.0
>            Reporter: Timothy St. Clair
> Modify Mesos Scheduler integration to allow SPARK_EXECUTOR_URI to be an executable script.
 This allows admins to launch spark in any fashion they desire, vs. just tarball fetching
+ implied context.   

This message was sent by Atlassian JIRA

View raw message