spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Taws (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property
Date Tue, 21 Jun 2016 08:24:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341360#comment-15341360
] 

Jonathan Taws edited comment on SPARK-15917 at 6/21/16 8:24 AM:
----------------------------------------------------------------

If I run the following command : {{spark-shell --master <master-url> --conf spark.cores.max=2
--conf spark.executor.instances=2 --executor-cores 2 --executor-memory 4g}}, I get only one
executor, as there are only 2 cores available (same with {{--num-executors}} instead of {{spark.executor.instances}}).


{{spark.cores.max}} does its job right here, and it just seems like the executor instances
property is just being discarded here. 

If anything, adding a support for a {{--num-executors}}-like property in standalone would
allow to be consistent across deployment methods and, in my experience, make the beginner
experience slightly easier. 


was (Author: jonathantaws):
If I run the following command : {{spark-shell \--master <master-url> \--conf spark.cores.max=2
\--conf spark.executor.instances=2 \--executor-cores 2 \--executor-memory 4g}}, I get only
one executor, as there are only 2 cores available (same with {{--num-executors}} instead of
{{spark.executor.instances}}). 

{{spark.cores.max}} does its job right here, and it just seems like the executor instances
property is just being discarded here. 

If anything, adding a support for a {{--num-executors}}-like property in standalone would
allow to be consistent across deployment methods and, in my experience, make the beginner
experience slightly easier. 

> Define the number of executors in standalone mode with an easy-to-use property
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-15917
>                 URL: https://issues.apache.org/jira/browse/SPARK-15917
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, Spark Shell, Spark Submit
>    Affects Versions: 1.6.1
>            Reporter: Jonathan Taws
>            Priority: Minor
>
> After stumbling across a few StackOverflow posts around the issue of using a fixed number
of executors in standalone mode (non-YARN), I was wondering if we could not add an easier
way to set this parameter than having to resort to some calculations based on the number of
cores and the memory you have available on your worker. 
> For example, let's say I have 8 cores and 30GB of memory available :
>  - If no option is passed, one executor will be spawned with 8 cores and 1GB of memory
allocated.
>  - However, if I want to have only *2* executors, and to use 2 cores and 10GB of memory
per executor, I will end up with *3* executors (as the available memory will limit the number
of executors) instead of the 2 I was hoping for.
> Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I want, but would
it not be easier to add a {{--num-executors}}-like option to standalone mode to be able to
really fine-tune the configuration ? This option is already available in YARN mode.
> From my understanding, I don't see any other option lying around that can help achieve
this.  
> This seems to be slightly disturbing for newcomers, and standalone mode is probably the
first thing anyone will use to just try out Spark or test some configuration.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message