mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Gupta <var...@uber.com>
Subject Re: Question on status update retry in agent
Date Mon, 09 Apr 2018 19:10:54 GMT
Hi,

We are running into an issue with slave status update manager. Below is the
behavior I am seeing.

Our use case is, we run Stateful container (Cassandra process), here
Executor polls JMX port at 60 second interval to get Cassandra State and
sends the state to agent -> master -> framework.

*RUNNING Cassandra Process translates to TASK_RUNNING.*
*CRASHED or DRAINED Cassandra Process translates to TASK_FAILED.*

At some point slave has multiple TASK_RUNNING status updates in stream and
then followed by TASK_FAILED if acknowledgements are pending. We use
explicit acknowledgements, and I see Mesos Master receives, all
TASK_RUNNING and then TASK_FAILED as well as Framework also receives all
TASK_RUNNING updates followed up TASK_FAILED. After receiving TASK_FAILED,
Framework restarts different executor on same machine using old persistent
volume.

Issue is that, *old executor reference is hold by slave* (assuming it did
not receive acknowledgement, whereas master and scheduler have processed
the status updates), so it  continues to retry TASK_RUNNING infinitely.
Here, old executor process is not running. As well as new executor process
is running, and continues to work as-is. This makes be believe, some bug
with slave status update manager.

I read slave status update manager code, recover
<https://github.com/apache/mesos/blob/master/src/slave/task_status_update_manager.cpp#L203>
 has a constraint
<https://github.com/apache/mesos/blob/master/src/slave/task_status_update_manager.cpp#L239>
 to ignore status updates from stream if the last executor run is completed.

I think, similar constraint should be applicable for status update
<https://github.com/apache/mesos/blob/master/src/slave/task_status_update_manager.cpp#L318>
 and acknowledge
<https://github.com/apache/mesos/blob/master/src/slave/task_status_update_manager.cpp#L760>
.


Thanks,
Varun


















On Fri, Mar 16, 2018 at 7:47 PM, Benjamin Mahler <bmahler@apache.org> wrote:

> (1) Assuming you're referring to the scheduler's acknowledgement of a
> status update, the agent will not forward TS2 until TS1 has been
> acknowledged. So, TS2 will not be acknowledged before TS1 is acknowledged.
> FWICT, we'll ignore any violation of this ordering and log a warning.
>
> (2) To reverse the question, why would it make sense to ignore them?
> Assuming you're looking to reduce the number of round trips needed for
> schedulers to see the terminal update, I would point you to:
> https://issues.apache.org/jira/browse/MESOS-6941
>
> (3) When the agent sees an executor terminate, it will transition all
> non-terminal tasks assigned to that executor to TASK_GONE (partition aware
> framework) or TASK_LOST (non partition aware framework) or TASK_FAILED (if
> the container OOMed). There may be other cases, it looks a bit convoluted
> to me.
>
> On Thu, Mar 15, 2018 at 10:35 AM, Zhitao Li <zhitaoli.cs@gmail.com> wrote:
>
> > Hi,
> >
> > While designing the correct behavior with one of our framework, we
> > encounters some questions about behavior of status update:
> >
> > The executor continuously polls the workload probe to get current mode of
> > workload (a Cassandra server), and send various status update states
> > (STARTING, RUNNING, FAILED, etc).
> >
> > Executor polls every 30 seconds, and sends status update. Here, we are
> > seeing congestion on task update acknowledgements somewhere (still
> > unknown).
> >
> > There are three scenarios that we want to understand.
> >
> >    1. Agent queue has task update TS1, TS2 & TS3 (in this order) waiting
> on
> >    acknowledgement. Suppose if TS2 receives an acknowledgement, then what
> > will
> >    happen to TS1 update in the queue.
> >
> >
> >    1. Agent queue has task update TS1, TS2, TS3 & TASK_FAILED. Here, TS1,
> >    TS2, TS3 are non-terminial updates. Once the agent has received a
> > terminal
> >    status update, does it makes sense to ignore non-terminal updates in
> the
> >    queue?
> >
> >
> >    1. As per Executor Driver code comment
> >    <https://github.com/apache/mesos/blob/master/src/java/src/
> > org/apache/mesos/ExecutorDriver.java#L86>,
> >    if the executor is terminated, does agent send TASK_LOST? If so, does
> it
> >    send once or for each unacknowledged status update?
> >
> >
> > I'll study the code in status update manager and agent separately but
> some
> > official answer will definitely help.
> >
> > Many thanks!
> >
> > --
> > Cheers,
> >
> > Zhitao Li
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message