spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (SPARK-26190) SparkLauncher: Allow users to set their own submitter script instead of hardcoded spark-submit
Date Tue, 27 Nov 2018 19:19:00 GMT


Marcelo Vanzin commented on SPARK-26190:

Based solely on what you wrote here, I'm leaning towards closing this.

SparkLauncher is a programatic API around spark-submit, not around your custom script. If
you have a custom script, you can call it without using SparkLauncher.

> SparkLauncher: Allow users to set their own submitter script instead of hardcoded spark-submit
> ----------------------------------------------------------------------------------------------
>                 Key: SPARK-26190
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Java API, Spark Core, Spark Submit
>    Affects Versions: 2.1.0
>         Environment: Apache Spark 2.0.1 on yarn cluster (MapR distribution)
>            Reporter: Gyanendra Dwivedi
>            Priority: Major
> Currently the script name is hard-coded in the 'createBuilder()' method of org.apache.spark.launcher.SparkLauncher
> private ProcessBuilder createBuilder() {
>  List<String> cmd = new ArrayList();
>  String script = CommandBuilderUtils.isWindows() ? "*spark-submit.cmd*" : "*spark-submit*";
>  cmd.add(CommandBuilderUtils.join(File.separator, new String[]\{this.builder.getSparkHome(),
"bin", script}));
>  cmd.addAll(this.builder.buildSparkSubmitArgs());
> ............
> .............
> }
> It has following issues, which prevents its usage in certain scenario. 
> 1) Developer may not use their own custom scripts with different name. They are forced
to use the one shipped with the installation. Overwriting that may not be the option, when
it is not allowed to alter the original installation.
> 2) The code expect the script to be present at "SPARK_HOME/bin" folder. 
> 3) The 'createBuilder()' method is private and hence, extending the 'org.apache.spark.launcher.SparkLauncher'
is not an option.
> Proposed solution:
> 1) Developer should be given option to set their own custom script, which may be located
at any path.
> 2) Only in case the parameter is not set, the default should be taken from SPARK_HOME/bin

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message