mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Timothy Chen (JIRA)" <>
Subject [jira] [Commented] (MESOS-1503) Improve slave health checking to prevent rapid widespread slave removals.
Date Sat, 21 Jun 2014 00:21:25 GMT


Timothy Chen commented on MESOS-1503:

I'm thinking about the design of the single slave observer, and here is my current thoughts:

1, Have a single SlavesObserver protobuf process, holds a map of all the slaves to be pinged,
a map of promises for the current ping, and current ping generation. 

It can have the following interface:

void registerSlave(SlaveID, UPID) : add a new slave to be health checked

void unregisterSlave(SlaveID) : remove slave to be health checked

void pingAllSlaves(): This sends out a ping to all slaves reigistered, which creates a promise
for each slave ping and holds it. In the end it collects all the from the promises futures
into one future and defer it to completePing(). It increments the generation id and sends
that via message body to all slaves.

void pong(UPID from, string body): response callback from slave ping. body is the current
ping generation id which the slave simply replies from the ping body. We also verify that
the pong is sent for the current ping generation, as if the pong is delayed and we received
an old pong we skip it. We also skip unregistered slave pongs too.

void timeout(UPID from): The timeout for the slave ping that just sets the promise to false.

void completePing(): In the end we look at all remaining futures and collect the failed ones,
verify that it still registered and send them all to the master for termination. We can either
have the opportunity to throttle or do more decisions based on all the failures at once. (We
can also move the logic to master, haven't really know what's best yet).

One issue came in mind is that now we're sending all the pings at once, and I wonder if it
can cause a burst of messages especially large amount of slaves. One way is to group slaves
to be pinged in different intervals, but could be something further in the future.

> Improve slave health checking to prevent rapid widespread slave removals.
> -------------------------------------------------------------------------
>                 Key: MESOS-1503
>                 URL:
>             Project: Mesos
>          Issue Type: Improvement
>          Components: master
>            Reporter: Benjamin Mahler
>            Assignee: Timothy Chen
>              Labels: reliability
> Per some discussions with [~tweingartner] and [~vinodkone].
> Currently the master uses a SlaveObserver for each registered slave. Each SlaveObserver
operates independently and makes decisions about whether the slave is healthy.
> The independence of these observers means that in some very rare events (e.g. masters
are partitioned from 75% of slaves), the master can very rapidly remove a large portion of
the slaves in the cluster. Ideally such an event could be deemed dangerous and throttled accordingly
through a more intelligent notion of overall cluster health.
> It may be nice to have a single observer that is responsible for health checking all
the slaves. This will allow us to make safer decisions as to when to determine that slaves
are unhealthy.

This message was sent by Atlassian JIRA

View raw message