Mailing-List: contact dev-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
MIME-Version: 1.0
In-Reply-To: <CAOOJoEy17wWpJ7j7k8WtkwwJyv0PdCX6_Pyo=KOD=YE-_-nnAg@mail.gmail.com>
References: <1326408418.5087954.1499988733637.ref@mail.yahoo.com>
 <1326408418.5087954.1499988733637@mail.yahoo.com> <CAOOJoEzmNDhrUkeojq_qaQxkptvsJnHqL8wSsoBZFN7XsFFR3A@mail.gmail.com>
 <4F8A36BA-2ABB-48F3-B95E-9B56E94248DA@yahoo.com> <CAOOJoEx4gOmbMRTRo9Hy4FUvZ4884rwnR7irC1B-gLfiU6zPfg@mail.gmail.com>
 <B56513D1-E638-49E6-AC80-DFB6EB7900EF@yahoo.com> <CAOOJoEywuVyUBr=iqY+t_EpT7TrCFPkxtUj3tOvgSR_2=GWjVw@mail.gmail.com>
 <CAOOJoExS7zfVrMSaEFK2LEvPL9Z-HSK3MFZBhW+_PR==PTP_OQ@mail.gmail.com>
 <C8F10450-7DA3-4169-BAF1-8FA99C0EB628@yahoo.com> <1500223744.32325.1.camel@stephanerb.eu>
 <782B3359-40EA-4C34-9696-1852F7DC7FC3@yahoo.com> <CAOOJoEy17wWpJ7j7k8WtkwwJyv0PdCX6_Pyo=KOD=YE-_-nnAg@mail.gmail.com>
From: Renan DelValle <rdelval1@binghamton.edu>
Date: Tue, 18 Jul 2017 10:44:59 -0700
Message-ID: <CAJ1-tocUimS3rD+5EoeipeP79RuhkcOKxC7W93YJ+wUDDjzgNQ@mail.gmail.com>
Subject: Re: Aurora reconciliation and Master fail over
To: David McLaughlin <david@dmclaughlin.com>
Cc: dev@aurora.apache.org
Content-Type: multipart/alternative; boundary="94eb2c1cd62a4844ee05549b14f5"
archived-at: Tue, 18 Jul 2017 17:45:33 -0000

--94eb2c1cd62a4844ee05549b14f5
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Yup, that looks like the way to go. Going to go ahead and file a ticket on
JIRA for this so that we don't forget. Thanks for digging into this David.

-Renan

On Mon, Jul 17, 2017 at 3:00 PM, David McLaughlin <david@dmclaughlin.com>
wrote:

> Based on the thread in the Mesos dev list, it looks like because they
> don't persist task information so they don't have the task IDs to send wh=
en
> they detect the agent is lost during failover. So unless this is changed =
on
> the Mesos side, we need to act on the slaveLost message and mark all thos=
e
> tasks as LOST in Aurora.
>
> Or rely on reconciliation. To reconcile more often, you should keep in
> mind:
>
> 1) Implicit reconciliation sends one message to Mesos and Mesos replies
> with N number of status updates immediately, where N =3D number of runnin=
g
> tasks. This process is usually quick (on the order of seconds) due to bei=
ng
> mostly NOOP status updates. When you have a large number of running tasks
> (say 100k+), you may see some GC pressure due to the flood of status
> updates. If this operation overlapped with another particularly expensive
> operation (like a snapshot) it can cause a huge stop the world GC. But it
> does not otherwise interfere with any operation.
>
> 2) Explicit reconciliation is done in batches, where Aurora batches up al=
l
> running tasks and sends one batch at a time, staggered by some delay. The
> benefit here is there is less GC pressure, but the drawback is if you hav=
e
> a lot of running tasks (again, 100k+), it will take over 10 minutes to
> complete. So you have to make sure your reconciliation interval is aligne=
d
> with this (you can always increase the batch size to make this happen
> faster).
>
> Cheers,
> David
>
> On Sun, Jul 16, 2017 at 11:10 AM, Meghdoot bhattacharya <
> meghdoot_b@yahoo.com.invalid> wrote:
>
>> Got it. Thx!
>>
>> > On Jul 16, 2017, at 9:49 AM, Stephan Erb <mail@stephanerb.eu> wrote:
>> >
>> > Reconciliation in Aurora is not a specific mode. It just runs
>> > concurrently to other background work such as snapshots or backups [1]=
.
>> >
>> >
>> > Just be aware that we don't have metrics to track the runtime of
>> > explicit and implicit reconciliations. If you use settings that are
>> > overly aggressive, you might overload Auroras queue of incoming Mesos
>> > status updates (for example).
>> >
>> > [1] https://github.com/apache/aurora/blob/c85bffdd6f68312261697eee868d=
5
>> > 7069adda434/src/main/java/org/apache/aurora/scheduler/reconciliation/T=
a
>> > skReconciler.java
>> >
>> >
>> >> On Sat, 2017-07-15 at 22:28 -0700, Meghdoot bhattacharya wrote:
>> >> Thx David for the follow up and confirmation.
>> >> We have started the thread on the mesos dev DL.
>> >>
>> >> So to get clarification on the recon, what is in general effect
>> >> during the recon. Does scheduling and activities like snapshot is
>> >> paused as recon takes place. Trying to see whether to run aggressive
>> >> recon in mean time.
>> >>
>> >> Thx
>> >>
>> >>> On Jul 15, 2017, at 9:33 AM, David McLaughlin <dmclaughlin@apache.o
>> >>> rg> wrote:
>> >>>
>> >>> I've left a comment on the initial RB detailing how the change
>> >>> broke
>> >>> backwards-compatibility. Given that the tasks are marked as lost as
>> >>> soon as
>> >>> the agent reregisters after slaveLost is sent anyway, there doesn't
>> >>> seem to
>> >>> be any reason not to send TASK_LOST too. I think this should be an
>> >>> easy
>> >>> fix.
>> >>>
>> >>> On Sat, Jul 15, 2017 at 9:21 AM, David McLaughlin <dmclaughlin@apac
>> >>> he.org>
>> >>> wrote:
>> >>>
>> >>>> Yes, we've confirmed this internally too (Santhosh did the work
>> >>>> here):
>> >>>>
>> >>>> When an agent becomes unreachable while the master is running, it
>> >>>> sends
>> >>>>> TASK_LOST events for each task on the agent.
>> >>>>> https://github.com/apache/mesos/blob/33093c893773f8c9d293afe
>> >>>>> 38e9909f9a2868d32/src/master/master.cpp#L7066-L7107
>> >>>>> Marking agent unreachable after failover does not cause
>> >>>>> TASK_LOST events.
>> >>>>> https://github.com/apache/mesos/blob/33093c893773f8c9d293afe
>> >>>>> 38e9909f9a2868d32/src/master/master.cpp#L2036-L2070
>> >>>>> Once an agent re-registers it sends TASK_LOST events. Agent
>> >>>>> sending
>> >>>>> TASK_LOST for tasks that it does not know after a Master
>> >>>>> failover.
>> >>>>> https://github.com/apache/mesos/blob/33093c893773f8c9d293afe
>> >>>>> 38e9909f9a2868d32/src/slave/slave.cpp#L1324-L1383
>> >>>>
>> >>>>
>> >>>>
>> >>>> The separate code path for markUnreachableAfterFailover appears
>> >>>> to have
>> >>>> been added by this commit:
>> >>>> https://github.com/apache/mesos/commit/937c85f2f6528d1ac56ea9a7aa
>> >>>> 174c
>> >>>> a0bd371d0c
>> >>>>
>> >>>> And I think this totally breaks the promise of introducing the
>> >>>> PARTITION_AWARE stuff in a backwards-compatible way.
>> >>>>
>> >>>> So right now, yes we rely on reconciliation to finally mark the
>> >>>> tasks as
>> >>>> LOST and reschedule their replacements.
>> >>>>
>> >>>> I think the only reason we haven't been more impacted by this at
>> >>>> Twitter
>> >>>> is our Mesos master is remarkably stable (compared to Aurora's
>> >>>> daily
>> >>>> failovers).
>> >>>>
>> >>>> We have two paths forward here: push forward and embrace the new
>> >>>> partition
>> >>>> awareness features in Aurora and/or push back on the above change
>> >>>> with the
>> >>>> Mesos community and have a better story for non-partition aware
>> >>>> APIs in the
>> >>>> short term.
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Sat, Jul 15, 2017 at 2:01 AM, Meghdoot bhattacharya <
>> >>>> meghdoot_b@yahoo.com.invalid> wrote:
>> >>>>
>> >>>>> We can reproduce it easily as the steps are
>> >>>>> 1. Shut down leading mesos master
>> >>>>> 2. Shutdown agent at same time
>> >>>>> 3. Wait for 10 mins
>> >>>>>
>> >>>>> What Renan and I saw in the logs were only agent lost and not
>> >>>>> task lost
>> >>>>> sent. While in regular health check expire scenario both task
>> >>>>> lost and
>> >>>>> agent lost were sent.
>> >>>>>
>> >>>>> So yes this is very concerning.
>> >>>>>
>> >>>>> Thx
>> >>>>>
>> >>>>>> On Jul 14, 2017, at 10:28 AM, David McLaughlin <dmclaughlin@a
>> >>>>>> pache.org>
>> >>>>>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> It would be interesting to see the logs. I think that will
>> >>>>>> tell you if
>> >>>>>
>> >>>>> the
>> >>>>>> Mesos master is:
>> >>>>>>
>> >>>>>> a) Sending slaveLost
>> >>>>>> b) Trying to send TASK_LOST
>> >>>>>>
>> >>>>>> And then the Scheduler logs (and/or the metrics it exports)
>> >>>>>> should tell
>> >>>>>
>> >>>>> you
>> >>>>>> whether those events were received. If this is reproducible,
>> >>>>>> I'd
>> >>>>>
>> >>>>> consider
>> >>>>>> it a serious bug.
>> >>>>>>
>> >>>>>> On Fri, Jul 14, 2017 at 10:04 AM, Meghdoot bhattacharya <
>> >>>>>> meghdoot_b@yahoo.com.invalid> wrote:
>> >>>>>>
>> >>>>>>> So in this situation why is not aurora replacing the tasks
>> >>>>>>> and waiting
>> >>>>>
>> >>>>> for
>> >>>>>>> external recon to fix it.
>> >>>>>>>
>> >>>>>>> This is different when the 75 sec (5*15) health check of
>> >>>>>>> slave times
>> >>>>>
>> >>>>> out
>> >>>>>>> (no master failover), aurora replaces it on task lost
>> >>>>>>> message.
>> >>>>>>>
>> >>>>>>> Are you hinting we should ask mesos folks why in master
>> >>>>>>> fail over
>> >>>>>>> reregistration timeout scenario why task lost not sent
>> >>>>>>> though slave
>> >>>>>
>> >>>>> lost
>> >>>>>>> sent and from below docs task lost should have been sent.
>> >>>>>>>
>> >>>>>>> Because either mesos is not sending the right status or
>> >>>>>>> aurora is not
>> >>>>>>> handling it.
>> >>>>>>>
>> >>>>>>> Thx
>> >>>>>>>
>> >>>>>>>> On Jul 14, 2017, at 8:21 AM, David McLaughlin <dmclaughli
>> >>>>>>>> n@apache.org
>> >>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>> "1. When mesos sends slave lost after 10 mins in this
>> >>>>>>>> situation , why
>> >>>>>>>
>> >>>>>>> does
>> >>>>>>>> aurora not act on it?"
>> >>>>>>>>
>> >>>>>>>> Because Mesos also sends TASK_LOST for every task running
>> >>>>>>>> on the agent
>> >>>>>>>> whenever it calls slaveLost:
>> >>>>>>>>
>> >>>>>>>> When it is time to remove an agent, the master removes
>> >>>>>>>> the agent from
>> >>>>>
>> >>>>> the
>> >>>>>>>> list of registered agents in the master=E2=80=99s durable state
>> >>>>>>>> <http://mesos.apache.org/documentation/latest/replicated-
>> >>>>>
>> >>>>> log-internals/>
>> >>>>>>> (this
>> >>>>>>>> will survive master failover). The master sends a
>> >>>>>>>> slaveLost callback
>> >>>>>
>> >>>>> to
>> >>>>>>>> every registered scheduler driver; it also sends
>> >>>>>>>> TASK_LOST status
>> >>>>>
>> >>>>> updates
>> >>>>>>>> for every task that was running on the removed agent.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Thu, Jul 13, 2017 at 4:32 PM, meghdoot bhattacharya <
>> >>>>>>>> meghdoot_b@yahoo.com.invalid> wrote:
>> >>>>>>>>
>> >>>>>>>>> We were investigation slave re registration behavior on
>> >>>>>>>>> master fail
>> >>>>>
>> >>>>> over
>> >>>>>>>>> in Aurora 0.17 with mesos 1.1.
>> >>>>>>>>> Few important points
>> >>>>>>>>> http://mesos.apache.org/documentation/latest/high-
>> >>>>>>>>> availability-framework-guide/ (If an agent does not
>> >>>>>>>>> reregister with
>> >>>>>
>> >>>>> the
>> >>>>>>>>> new master within a timeout (controlled by the
>> >>>>>>>
>> >>>>>>> --agent_reregister_timeout
>> >>>>>>>>> configuration flag), the master marks the agent as
>> >>>>>>>>> failed and follows
>> >>>>>>>
>> >>>>>>> the
>> >>>>>>>>> same steps described above. However, there is one
>> >>>>>>>>> difference: by
>> >>>>>>>
>> >>>>>>> default,
>> >>>>>>>>> agents are allowed to reconnect following master
>> >>>>>>>>> failover, even after
>> >>>>>>>
>> >>>>>>> the
>> >>>>>>>>> agent_reregister_timeout has fired. This means that
>> >>>>>>>>> frameworks might
>> >>>>>>>
>> >>>>>>> see a
>> >>>>>>>>> TASK_LOST update for a task but then later discover
>> >>>>>>>>> that the task is
>> >>>>>>>>> running (because the agent where it was running was
>> >>>>>>>>> allowed to
>> >>>>>>>
>> >>>>>>> reconnect).
>> >>>>>>>>> http://mesos.apache.org/documentation/latest/reconcilia
>> >>>>>>>>> tion/
>> >>>>>
>> >>>>> (Implicit
>> >>>>>>>>> reconciliation (passing an empty list) should also be
>> >>>>>>>>> used
>> >>>>>>>
>> >>>>>>> periodically, as
>> >>>>>>>>> a defense against data loss in the framework. Unless a
>> >>>>>>>>> strict
>> >>>>>
>> >>>>> registry
>> >>>>>>> is
>> >>>>>>>>> in use on the master, its possible for tasks to
>> >>>>>>>>> resurrect from a LOST
>> >>>>>>>
>> >>>>>>> state
>> >>>>>>>>> (without a strict registry the master does not enforce
>> >>>>>>>>> agent removal
>> >>>>>>>
>> >>>>>>> across
>> >>>>>>>>> failovers). When an unknown task is encountered, the
>> >>>>>>>>> scheduler should
>> >>>>>>>
>> >>>>>>> kill
>> >>>>>>>>> or recover the task.)
>> >>>>>>>>> https://issues.apache.org/jira/browse/MESOS-5951(Remove
>> >>>>>>>>> s strict
>> >>>>>>>
>> >>>>>>> registry
>> >>>>>>>>> mode flag from 1.1 and reverts to the old behavior of
>> >>>>>>>>> non strict
>> >>>>>>>
>> >>>>>>> registry
>> >>>>>>>>> mode where tasks and executors were not killed on agent
>> >>>>>
>> >>>>> reregistration
>> >>>>>>>>> timeout on master failover)
>> >>>>>>>>> So, what we find, if the slave does not come back after
>> >>>>>>>>> 10 mins
>> >>>>>>>>> 1. Mesos master sends slave lost but not task lost to
>> >>>>>>>>> Aurora.2.
>> >>>>>
>> >>>>> Aurora
>> >>>>>>>>> does not replace the tasks.3. When explicit recon
>> >>>>>>>>> starts , then only
>> >>>>>>>
>> >>>>>>> this
>> >>>>>>>>> gets corrected with aurora spawning replacement tasks.
>> >>>>>>>>> If slave restarts after 10 mins
>> >>>>>>>>> 1. When implicit recon starts, this situation gets
>> >>>>>>>>> fixed because in
>> >>>>>>>
>> >>>>>>> aurora
>> >>>>>>>>> it is marked as lost and mesos sends running and those
>> >>>>>>>>> get killed and
>> >>>>>>>>> replaced.
>> >>>>>>>>> So, questions
>> >>>>>>>>> 1. When mesos sends slave lost after 10 mins in this
>> >>>>>>>>> situation , why
>> >>>>>>>
>> >>>>>>> does
>> >>>>>>>>> aurora not act on it?2. As per recon docs best
>> >>>>>>>>> practices, explicit
>> >>>>>
>> >>>>> recon
>> >>>>>>>>> should start followed by implicit recon on master
>> >>>>>>>>> failover. Looks
>> >>>>>
>> >>>>> like
>> >>>>>>>>> aurora is not doing that and the regular hourly recons
>> >>>>>>>>> are running
>> >>>>>
>> >>>>> with
>> >>>>>>> 30
>> >>>>>>>>> min spread between explicit and implicit. Should aurora
>> >>>>>>>>> do recon on
>> >>>>>>>
>> >>>>>>> master
>> >>>>>>>>> fail over?
>> >>>>>>>>>
>> >>>>>>>>> General questions1. What is the effect on aurora if we
>> >>>>>>>>> make explicit
>> >>>>>>>
>> >>>>>>> recon
>> >>>>>>>>> every 15 mins instead of default 1 hr? Does it slow
>> >>>>>>>>> down scheduling,
>> >>>>>>>
>> >>>>>>> does
>> >>>>>>>>> snapshot creation gets delayed etc?
>> >>>>>>>>> 2. Any issue if spread between explicit recon and
>> >>>>>>>>> implicit recon
>> >>>>>
>> >>>>> brought
>> >>>>>>>>> down to 2 mins from 30 mins? probably depend on 1.
>> >>>>>>>>> Thx
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>>
>> >>
>> >>
>>
>>
>

--94eb2c1cd62a4844ee05549b14f5--