Mailing-List: contact dev-help@aurora.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAOTkfX4KTUpMVcjeFf5=vvGXb91to5baNSzvyiwtk-sTddxGXQ@mail.gmail.com>
References: 
 <CAEYWR9vt1WBV6nq9RSRt36i1ZmPFNiXGeB+_L2SviYe_tsQSug@mail.gmail.com>
	<CAEYWR9tKM1vWjRrb+MOB9AnB7wHjxL8rkzgyF_YyYV1Qe6XRmw@mail.gmail.com>
	<CAEYWR9sedw0N3Yv45ZyOY4yEsoUYFqV=LOaoGtTGCp+ksJ+Wcg@mail.gmail.com>
	<CAOTkfX4KTUpMVcjeFf5=vvGXb91to5baNSzvyiwtk-sTddxGXQ@mail.gmail.com>
Date: Thu, 18 Dec 2014 11:56:44 -0800
Message-ID: 
 <CAAATh-bP6yUpiGxO29c5KUU6rsG4mU_OzHfLrHVmSgQpzQ=HDw@mail.gmail.com>
Subject: Re: aurora watch_secs change
From: Kevin Sweeney <kevints@apache.org>
To: Aurora <dev@aurora.incubator.apache.org>
Content-Type: multipart/alternative; boundary=047d7bb708c2de992d050a82fd8d

--047d7bb708c2de992d050a82fd8d
Content-Type: text/plain; charset=UTF-8

On Wed, Dec 17, 2014 at 4:33 PM, Maxim Khutornenko <maxim@apache.org> wrote:
>
> Here is in-person discussion follow up. Participants: Moses, wickman,
> kevints, maxim.
>
> The proposal we came up with does not require implementing scheduler
> health checks (AURORA-279). The idea is to require the executor to
> move a task from STARTING to RUNNING only when its health checks are
> satisfied. This will make the updater go faster by relying directly on
> RUNNING status update, which is now going to be a true reflection of a
> healthy user task. The watch_secs will still be useful for updating
> tasks without the health checks enabled.
>
> Below is a high level summary of required changes (incomplete).
>
> Scheduler:
> - Modify task state machine to treat STARTING as a new active
> (non-transient) state
> - Modify Preemptor to account for STARTING
> - Modify stats and SLA metrics to properly account for STARTING
> - Modify scheduler updater to short-circuit watch_secs when health
> checks are enabled
>
I don't think this is necessary (it'd require teaching the scheduler about
thermos health checks). When using active health checks you can set
watch_secs to 0.


>
> Schema:
> - Add max_consecutive_successes setting into HealthCheckConfig [1] to
> instruct the executor when to move task into RUNNING.
>
> Executor:
> - Modify state transition logic to rely on health checks (if enabled)
> to move the task into RUNNING. Transition from STARTING to RUNNING
> immediately if task health checks are disabled.
>
> Open question: with STARTING becoming a non-transient state from the
> scheduler standpoint, there is nothing to enforce its exit. This may
> be OK as STARTING will effectively be a stable user defined state.
> However, this is something we may want to cap to avoid adverse user
> impact.
>
> Thoughts?
>
> Thanks,
> Maxim
>
> [1] -
> https://github.com/apache/incubator-aurora/blob/master/docs/configuration-reference.md#healthcheckconfig-objects
>
> On Sat, Dec 13, 2014 at 11:06 AM, Nakamura <nnythm@gmail.com> wrote:
> > Hey,
> > Just wanted to make sure my email didn't get lost in the cracks.
> >
> > As a reminder, the previous emails in this thread were:
> > Bill Farner
> > <
> http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201412.mbox/ajax/%3CCAGRA8uMpWyhcV-hxLU%3Dw7twDD7jbffu39TmbX5MPiXQE8jextA%40mail.gmail.com%3E
> >
> > Brian Wickman
> > <
> http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201412.mbox/ajax/%3CCAFTdr0DerXKtK%2BhGrJDN0VU-RgQ8sisCKaAZ3Jzg11BTzea5gw%40mail.gmail.com%3E
> >
> >
> > Best,
> > Moses
> >
> > On Thu Dec 04 2014 at 11:14:02 AM Nakamura <nnythm@gmail.com> wrote:
> >
> >> Hey,
> >>
> >> Sorry that this is replying to my own email, I didn't realize that I had
> >> to subscribe to the dev@aurora listserv to get updates.  This email
> >> should really be in response to Brian Wickman's response.
> >>
> >> Hmm, I don't think only sending the transitions is sufficient though.
> My
> >> concern is that since sending framework messages isn't reliable, we
> could
> >> end up in a situation where the scheduler perceives the task is healthy
> >> even though it's not.
> >>
> >> 1. scheduler spins up executor
> >> 2. executor unhealthy
> >> 3. executor transitions to healthy, sends message to scheduler
> >> 4. scheduler receives healthy message
> >> 5. executor transitions to unhealthy before N healthy messages, sends
> >> message to scheduler
> >> 6. scheduler does not receive unhealthy message
> >> 7. after waiting for N messages * time between messages without a
> >> response, it assumes that it has remained healthy and marks it as
> healthy
> >> enough to continue.
> >>
> >> We can fix this by changing 7 to include the check that's currently
> >> included in the watch_secs delayed action.
> >>
> >> Here is my new proposal for how B should work:
> >>
> >> Executor sends health transitions as framework messages to the
> >> scheduler.  When the scheduler receives a transition to healthiness, it
> >> waits for N messages * time between messages, and then sends a request
> to
> >> ask if the executor is still healthy.  If the scheduler never sees a
> >> healthy message, it defaults to the old behavior, sending a request at
> >> watch_secs. Once the scheduler no longer needs the transitions, it tells
> >> the executor to stop sending the messages.
> >>
> >> Thoughts?  Are there any easy ways I can simplify the design?
> >>
> >> Best,
> >> Moses
> >>
> >> On Tue Dec 02 2014 at 1:53:24 PM Nakamura <nnythm@gmail.com> wrote:
> >>
> >>> Howdy,
> >>>
> >>> I'm interested in tackling AURORA-894, but I'm not terribly familiar
> with
> >>> aurora, so I'd like some feedback on my design before I go forth.
> >>>
> >>> Bill pointed out that the hard bit would be designing the algorithm so
> it
> >>> doesn't DDoS the scheduler, and I think I have an idea of the possible
> >>> design space.  I wanted to know what you thought.
> >>>
> >>> A.  sample the number of health checks, and send them back to the
> >>> scheduler.  this is pretty simple, but 99% of the time will be total
> noise,
> >>> since the data isn't generally useful.
> >>>
> >>> B.  the executor sends health checks until it receives an out of band
> >>> request from the scheduler not to.  this seems fragile (I'm imagining
> >>> mismatched executors/schedulers behaving poorly) but would also
> probably be
> >>> reasonably simple.
> >>>
> >>> C.  a slightly more sophisticated approach might be to tell the
> executor
> >>> how many health checks to look for, so that it could send a status
> update
> >>> back, since status updates have reliable delivery.
> >>>
> >>> D. when the scheduler has finished standing up the executor, it
> >>> long-polls, which also takes care of reliable delivery because it's
> >>> presumably over TCP and we have total control (not having to go through
> >>> mesos).
> >>>
> >>> I'm hesitant to do A, because it's so wasteful.  B sounds fragile, so I
> >>> don't want to do that one.  D requires long-polling, which your client
> may
> >>> or may not do well.  I'm leaning toward C.  Do you think that sounds
> like a
> >>> reasonable approach?
> >>>
> >>> Thanks,
> >>> Moses
> >>>
> >>
>

--047d7bb708c2de992d050a82fd8d--