mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kone (JIRA)" <>
Subject [jira] [Assigned] (MESOS-8209) mesos master should revoke offers when executor state changes
Date Mon, 13 Nov 2017 19:59:00 GMT


Vinod Kone reassigned MESOS-8209:

    Assignee: Vinod Kone

> mesos master should revoke offers when executor state changes
> -------------------------------------------------------------
>                 Key: MESOS-8209
>                 URL:
>             Project: Mesos
>          Issue Type: Bug
>            Reporter: Jack Crawford
>            Assignee: Vinod Kone
> Currently, the mesos master does not revoke offers when the number of executors on an
agent decreases. This is a problem under certain conditions, such when running a workflow
that starts lots of small tasks on agents, with a one executor per task model, a master that
does not revoke resources after a set amount of time, and a scheduler that does not reject
> The problem is that when running a mono-scheduler framework (which you might want to
do to easily enforce authentication requirements, have a full view of all scheduled tasks,
etc), in order to respond instantly when new tasks come in I have the scheduler simply hang
on to all resource offers it receives, and the master is set to never revoke offers. This
way the scheduler always has a pool of resources to quickly service new requests as they come
> However, if you start tasks fast enough, the agents can fill up with executors, making
it appear as there are no resources available for the scheduler to use. Ive seen this on r4.4xlarge
machines on aws with executors that consume 0.1 cpus, 32mb mem where the entire machine will
be appear to be filled with executors according to the master resource offers. The executors
are exiting (just after the task finishes), but the resources are not reclaimed because the
master does not revoke the outstanding resource offers to reflect the change.
> You can replicate this pretty easily if you schedule tasks that finish instantly with
a 1-1 executor to task ratio. I find that if I schedule ~1000 tasks this way on a single r4.4xlarge
machine, usually 600-700 will finish before all the resource offers to the scheduler fill
up and the agent appears to be "full" of executors.
> Changing the scheduler/master to periodically reject/revoke resources fixes the problem.
> My suggestion is for the master to revoke and reissue resource offers when the executor
count changes on an agent.

This message was sent by Atlassian JIRA

View raw message