mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kone (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-1688) No offers if no memory is allocatable
Date Tue, 26 Aug 2014 18:45:58 GMT

    [ https://issues.apache.org/jira/browse/MESOS-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111125#comment-14111125
] 

Vinod Kone commented on MESOS-1688:
-----------------------------------

We can change the allocator to send offers with only cpus or only memory but we need to make
sure frameworks DO NOT launch executors with only cpus or only memory. This is because it
doesn't make sense to launch an executor with only cpus/memory (esp. when there are no tasks
running on it).

This is something we've been talking for a while and we could add a check in the master to
reject such task/executor launches. This likely needs a heads up to the dev list in case there
are any frameworks out there that launch such executors.

> No offers if no memory is allocatable
> -------------------------------------
>
>                 Key: MESOS-1688
>                 URL: https://issues.apache.org/jira/browse/MESOS-1688
>             Project: Mesos
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1
>            Reporter: Martin Weindel
>            Priority: Critical
>
> The [Spark scheduler|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala]
allocates memory only for the executor and cpu only for its tasks.
> So it can happen that all memory is nearly completely allocated by Spark executors, but
all cpu resources are idle.
> In this case Mesos does not offer resources anymore, as less than MIN_MEM (=32MB) memory
is allocatable.
> This effectively causes a dead lock in the Spark job, as it is not offered cpu resources
needed for launching new tasks.
> see {{HierarchicalAllocatorProcess::allocatable(const Resources&)}} called in {{HierarchicalAllocatorProcess::allocate(const
hashset<SlaveID>&)}}
> {code}
> template <class RoleSorter, class FrameworkSorter>
> bool
> HierarchicalAllocatorProcess<RoleSorter, FrameworkSorter>::allocatable(
>     const Resources& resources)
> {
> ...
>   Option<double> cpus = resources.cpus();
>   Option<Bytes> mem = resources.mem();
>   if (cpus.isSome() && mem.isSome()) {
>     return cpus.get() >= MIN_CPUS && mem.get() > MIN_MEM;
>   }
>   return false;
> }
> {code}
> A possible solution may to completely drop the condition on allocatable memory.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message