spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Assigned] (SPARK-13723) YARN - Change behavior of --num-executors when spark.dynamicAllocation.enabled true
Date Thu, 26 May 2016 22:44:13 GMT


Apache Spark reassigned SPARK-13723:

    Assignee: Apache Spark

> YARN - Change behavior of --num-executors when spark.dynamicAllocation.enabled true
> -----------------------------------------------------------------------------------
>                 Key: SPARK-13723
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 2.0.0
>            Reporter: Thomas Graves
>            Assignee: Apache Spark
>            Priority: Minor
> I think we should change the behavior when --num-executors is specified when dynamic
allocation is enabled. Currently if --num-executors is specified dynamic allocation is disabled
and it just uses a static number of executors.
> I would rather see the default behavior changed in the 2.x line. If dynamic allocation
config is on then num-executors goes to max and initial # of executors. I think this would
allow users to easily cap their usage and would still allow it to free up executors. It would
also allow users doing ML start out with a # of executors and if they are actually caching
the data the executors wouldn't be freed up. So you would get very similar behavior to if
dynamic allocation was off.
> Part of the reason for this is when using a static number if generally wastes resources,
especially with people doing adhoc things with spark-shell. It also has a big affect when
people are doing MapReduce/ETL type work loads.   The problem is that people are used to specifying
num-executors so if we turn it on by default in a cluster config its just overridden.
> We should also update the spark-submit --help description for --num-executors

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message