spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ash (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-1882) Support dynamic memory sharing in Mesos
Date Fri, 09 Jan 2015 05:04:34 GMT

    [ https://issues.apache.org/jira/browse/SPARK-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270559#comment-14270559
] 

Andrew Ash commented on SPARK-1882:
-----------------------------------

Yes -- in the simplest case where you have one machine with one job running with all resources
and then a second job starts, you'd want the first job to yield half its resources and then
the two jobs both have half the memory and cores of the box.  So you'd need to change memory
size while running jobs.

That said, I'm still new to this resource management world and trying to understand if this
is actually an issue in practice or not -- the recent work of having dynamic scaling with
YARN and the separate shuffle server seems like it has the core sharing figured out, but from
my memory of watching this video I'm not sure that it has the memory sharing worked out. 
https://www.youtube.com/watch?v=yxcc8fCbjQI

> Support dynamic memory sharing in Mesos
> ---------------------------------------
>
>                 Key: SPARK-1882
>                 URL: https://issues.apache.org/jira/browse/SPARK-1882
>             Project: Spark
>          Issue Type: Improvement
>          Components: Mesos
>    Affects Versions: 1.0.0
>            Reporter: Andrew Ash
>
> Fine grained mode Mesos currently supports sharing CPUs very well, but requires that
memory be pre-partitioned according to the executor memory parameter.  Mesos supports dynamic
memory allocation in addition to dynamic CPU allocation, so we should utilize this feature
in Spark.
> See below where when the Mesos backend accepts a resource offer it only checks that there's
enough memory to cover sc.executorMemory, and doesn't ever take a fraction of the memory available.
 The memory offer is accepted all or nothing from a pre-defined parameter.
> Coarse mode:
> https://github.com/apache/spark/blob/3ce526b168050c572a1feee8e0121e1426f7d9ee/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala#L208
> Fine mode:
> https://github.com/apache/spark/blob/a5150d199ca97ab2992bc2bb221a3ebf3d3450ba/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala#L114



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message