spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Imran Rashid (JIRA)" <>
Subject [jira] [Commented] (SPARK-14822) Add lazy executor startup to Mesos Scheduler
Date Thu, 14 Dec 2017 19:27:01 GMT


Imran Rashid commented on SPARK-14822:

[~mgummelt] is this still relevant?  seems like dynamic allocation on mesos already covers
this (not precisely the same but pretty close)

> Add lazy executor startup to Mesos Scheduler
> --------------------------------------------
>                 Key: SPARK-14822
>                 URL:
>             Project: Spark
>          Issue Type: Task
>          Components: Mesos
>            Reporter: Michael Gummelt
> As we deprecate fine-grained mode, we need to make sure we have alternative solutions
for its benefits.
> Its two benefits are:
> 0. lazy executor startup
>   In fine-grained mode, executors are brought up only as tasks are scheduled.  This means
that a user doesn't have to set {{spark.cores.max}} to ensure that the app doesn't consume
all resources in the cluster.
> 1. relinquishing cores
>   As Spark tasks terminate, the mesos task it was bound to terminates as well, thus relinquishing
the cores assigned to it.
> I'd like to add {{0.}} to coarse-grained mode, possibly enabled with a configuration
param.  If ever happens, we can add {{1.}}
as well.
> cc [~tnachen] [~dragos] [~skonto] [~andrewor14]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message