aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Farner <wfar...@apache.org>
Subject Re: Heartbeat mechanism auditing
Date Fri, 30 Jan 2015 00:12:59 GMT
I'm actually beginning to think that an explicit state for "waiting for a
heartbeat" might be easier to implement than volatile state.  In a world
where job updates are fully automated, i could see a bunch of users asking
why a job update made no progress for a period of time, so it's really nice
if the administrator doesn't need to dig into logs to piece history back
together.

Maxim - can you elaborate on an example of the complexity you are concerned
about?


-=Bill

On Thu, Jan 29, 2015 at 3:52 PM, Bill Farner <wfarner@apache.org> wrote:

> Here's the permalink to the thread in question:
> http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201410.mbox/%3CCAOTkfX7x2oipk4ZFysoS0uWZRizOnKJA3y15pvEW5K4YnUHw-A%40mail.gmail.com%3E
>
> -=Bill
>
> On Thu, Jan 29, 2015 at 2:45 PM, Maxim Khutornenko <maxim@apache.org>
> wrote:
>
>> To add a bit of history to the topic, the current design has been
>> debated heavily here [1] and an active/lazy consensus was reached
>> around implementing the first iteration as lightweight as possible
>> without persisting any durable state.
>>
>> My take on this - we should proceed as originally proposed given the
>> following:
>>
>> - History of heartbeats is the only feature that requires state
>> persistence. Nothing else in the current design benefits from
>> persisting the state across restarts. I consider pulse history as a
>> nice to have rather than a requirement (unlike the current state
>> reporting, which is a must for troubleshooting and is racked by
>> AURORA-1049).
>>
>> - State persistence will come with additional complexity of handling
>> corner cases (restart, abort, resume, etc.) that is not well justified
>> at this point given our total lack of experience with heartbeats.
>>
>> - Adding pulse history tracking can be done at later stages (as the
>> feature evolves and we gain more insight) without the adverse user
>> impact or technical debt. On the contrary, if attempted early the
>> overlooked details may hurt down the road by requiring Thrift schema
>> migration.
>>
>> Thanks,
>> Maxim
>>
>> [1] -
>> http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201410.mbox/browser
>>
>> On Thu, Jan 29, 2015 at 2:07 PM, David McLaughlin
>> <dmclaughlin@apache.org> wrote:
>> > Hi all,
>> >
>> > There is a little bit of a stalemate with regards to the implementation
>> of
>> > the pulse RPC in the scheduler.
>> >
>> > As a brief overview of this feature - the pulse RPC is designed so that
>> an
>> > external service can monitor the new in-scheduler updates reliably. This
>> > external service could be doing something like keeping an eye on
>> > application level alerts and pausing the update if things slip into a
>> bad
>> > state. The purpose of the pulse is to make sure the update does not
>> > continue if it's not being monitored (i.e. the external service might
>> have
>> > failed) by requiring positive acknowledgement at a given time interval.
>> >
>> > The implementation is in this review:
>> https://reviews.apache.org/r/30225/
>> >
>> > The contention is around whether or not the "blocked" state deserves its
>> > own explicit state in the update state machine, and whether this is
>> > important enough to block the review. Currently any blocked updates are
>> > only known to the scheduler and the update will show as
>> > UPDATING/ROLLING_FORWARD in the UI and any history that the update was
>> > blocked will be lost - we only track current state.
>> >
>> > If you have any opinions on this feature, please feel free to chime in
>> to
>> > the RB!
>> >
>> > Thanks,
>> > David
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message