spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Imran Rashid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-14822) Add lazy executor startup to Mesos Scheduler
Date Thu, 14 Dec 2017 19:27:01 GMT

    [ https://issues.apache.org/jira/browse/SPARK-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291404#comment-16291404
] 

Imran Rashid commented on SPARK-14822:
--------------------------------------

[~mgummelt] is this still relevant?  seems like dynamic allocation on mesos already covers
this (not precisely the same but pretty close)

> Add lazy executor startup to Mesos Scheduler
> --------------------------------------------
>
>                 Key: SPARK-14822
>                 URL: https://issues.apache.org/jira/browse/SPARK-14822
>             Project: Spark
>          Issue Type: Task
>          Components: Mesos
>            Reporter: Michael Gummelt
>
> As we deprecate fine-grained mode, we need to make sure we have alternative solutions
for its benefits.
> Its two benefits are:
> 0. lazy executor startup
>   In fine-grained mode, executors are brought up only as tasks are scheduled.  This means
that a user doesn't have to set {{spark.cores.max}} to ensure that the app doesn't consume
all resources in the cluster.
> 1. relinquishing cores
>   As Spark tasks terminate, the mesos task it was bound to terminates as well, thus relinquishing
the cores assigned to it.
> I'd like to add {{0.}} to coarse-grained mode, possibly enabled with a configuration
param.  If https://issues.apache.org/jira/browse/MESOS-1279 ever happens, we can add {{1.}}
as well.
> cc [~tnachen] [~dragos] [~skonto] [~andrewor14]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message