spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Cuquemelle (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number
Date Wed, 06 Dec 2017 17:33:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280548#comment-16280548
] 

Julien Cuquemelle edited comment on SPARK-22683 at 12/6/17 5:32 PM:
--------------------------------------------------------------------

I don't understand your statement about delaying executor addition ? I want to cap the number
of executors in an adaptive way regarding the current number of tasks, not delay their creation.

Doing this with dynamicAllocation.maxExecutors requires each job to be tuned for efficiency;
when we're doing experiments, a lot of jobs are one shot, so they can't be fine tuned.
The proposal gives a way to have an adaptive behaviour for a family of jobs.

Regarding slowing the ramp up of executors with schedulerBacklogTimeout, I've made experiments
to play with this parameter; I have made 2 series of experiments (7 jobs on each test case,
average figures reported in the following table), one on a busy queue, and the other on an
idle queue. I'll report only the idle queue, as the figures 
on the busy queue are even worse for the schedulerBacklogTimeout approach: 

First row is using the default 1s for the schedulerBacklogTimeout, and uses the 6 tasks per
executorSlot I've mentioned above, other rows use the default dynamicAllocation behaviour
and only change schedulerBacklogTimeout

||SparkWallTimeSec||Spk-vCores-H||taskPerExeSlot||schedulerBacklogTimeout||
|693.571429|37.142857|6|1.0|
|584.857143|69.571429|1|30.0|
|763.428571|54.285714|1|60.0|
|826.714286|39.571429|1|90.0|

So basically I can tune the backlogTimeout to get a similar vCores-H consumption at the expense
of almost 20% more wallClockTime, or I can tune the parameter to get about the same wallClockTime
at the expense of about 60% more vcoreH consumption (very roughly extrapolated between 30
and 60 secs for schedulerBacklogTimeout).

It does not seem to solve the issue I'm trying to address, moreover this would again need
to be tuned for each specific job's duration (to find the 90s timeout to get the similar resource
consumption, I had to solve the exponential ramp-up with the duration of the already run job,
which is not feasible in experimental use cases ).
The previous experiments that allowed me to find the sweet spot at 6 tasks per slot has involved
job wallClockTimes between 400 and 9000 seconds

Another way to have a look at this new parameter I'm proposing is to have a simple way to
tune the latency / resource consumption tradeoff. 


was (Author: jcuquemelle):
I don't understand your statement about delaying executor addition ? I want to cap the number
of executors in an adaptive way regarding the current number of tasks, not delay their creation.

Doing this with dynamicAllocation.maxExecutors requires each job to be tuned for efficiency;
when we're doing experiments, a lot of jobs are one shot, so they can't be fine tuned.
The proposal gives a way to have an adaptive behaviour for a family of jobs.

Regarding slowing the ramp up of executors with schedulerBacklogTimeout, I've made experiments
to play with this parameter; I have made 2 series of experiments (7 jobs on each test case,
average figures reported in the following table), one on a busy queue, and the other on an
idle queue. I'll report only the idle queue, as the figures 
on the busy queue are even worse for the schedulerBacklogTimeout approach: 

First row is using the default 1s for the schedulerBacklogTimeout, and uses the 6 tasks per
executorSlot I've mentioned above, other rows use the default dynamicAllocation behaviour
and only change schedulerBacklogTimeout

SparkWallTimeSec    Spk-vCores-H    taskPerExeSlot    schedulerBacklogTimeout

    693.571429            37.142857          6                         1.0
    584.857143            69.571429          1                         30.0
    763.428571            54.285714          1                         60.0
    826.714286            39.571429          1                         90.0

So basically I can tune the backlogTimeout to get a similar vCores-H consumption at the expense
of almost 20% more wallClockTime, or I can tune the parameter to get about the same wallClockTime
at the expense of about 60% more vcoreH consumption (very roughly extrapolated between 30
and 60 secs for schedulerBacklogTimeout).

It does not seem to solve the issue I'm trying to address, moreover this would again need
to be tuned for each specific job's duration (to find the 90s timeout to get the similar resource
consumption, I had to solve the exponential ramp-up with the duration of the already run job,
which is not feasible in experimental use cases ).
The previous experiments that allowed me to find the sweet spot at 6 tasks per slot has involved
job wallClockTimes between 400 and 9000 seconds

Another way to have a look at this new parameter I'm proposing is to have a simple way to
tune the latency / resource consumption tradeoff. 

> Allow tuning the number of dynamically allocated executors wrt task number
> --------------------------------------------------------------------------
>
>                 Key: SPARK-22683
>                 URL: https://issues.apache.org/jira/browse/SPARK-22683
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.1.0, 2.2.0
>            Reporter: Julien Cuquemelle
>              Labels: pull-request-available
>
> let's say an executor has spark.executor.cores / spark.task.cpus taskSlots
> The current dynamic allocation policy allocates enough executors
> to have each taskSlot execute a single task, which minimizes latency, 
> but wastes resources when tasks are small regarding executor allocation
> overhead. 
> By adding the tasksPerExecutorSlot, it is made possible to specify how many tasks
> a single slot should ideally execute to mitigate the overhead of executor
> allocation.
> PR: https://github.com/apache/spark/pull/19881



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message