aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Farner <wfar...@apache.org>
Subject Re: [PROPOSAL] Job instance scaling APIs
Date Wed, 20 Jan 2016 01:02:52 GMT
Nit: your examples above include SessionKey, which no longer exists.

Otherwise LGTM.

On Tue, Jan 19, 2016 at 4:58 PM, Maxim Khutornenko <maxim@apache.org> wrote:

> I think the behavior is spot on.
>
> So the plan is to move the current:
>
> Response addInstances(
>     1: AddInstancesConfig config,
>     2: Lock lock,
>     3: SessionKey session)
>
> to:
>
> Response addInstances(
>     1: AddInstancesConfig config,
>     2: Lock lock,
>     3: SessionKey session,
>     4: InstanceKey instanceKey,
>     5: int32 instanceCount)
>
> In the current release we will fork the behavior depending on whether
> the InstanceKey is present. In the next release, we drop 1,2 and 3 and
> end up with only 4 and 5.
>
> Any concerns?
>
> On Tue, Jan 19, 2016 at 3:57 PM, Bill Farner <wfarner@apache.org> wrote:
> > Unless the behavior differs significantly, repurposing sounds like the
> > right thing to do.  We don't *need* a separate RPC to accomplish this, we
> > can just add arguments or struct fields to accomplish dual use.
> >
> > On Tue, Jan 19, 2016 at 3:53 PM, Maxim Khutornenko <maxim@apache.org>
> wrote:
> >
> >> That thought crossed my mind but due to addInstances() being reserved
> >> for an existing RPC I filed AURORA-1581 to handle its graceful
> >> deprecation first.
> >>
> >> Bill, are you suggesting re-purposing the existing 'addInstances()'
> >> into a 'scaleOut()' equivalent? I am mostly +1 on the change but if we
> >> do it within the same release we will not be following our deprecation
> >> guidelines. Perhaps we can name it something like 'addTaskInstances()'
> >> instead and let addInstances() go away naturally?
> >>
> >> On Tue, Jan 19, 2016 at 3:06 PM, Tony Dong <tdong@twitter.com.invalid>
> >> wrote:
> >> > +1 to addInstances
> >> >
> >> > On Tue, Jan 19, 2016 at 3:00 PM, Bill Farner <wfarner@apache.org>
> wrote:
> >> >
> >> >> At risk of devolving the discussion, is it worth calling the method
> >> >> addInstances as opposed to scaleOut?  I find the former more
> >> descriptive.
> >> >>
> >> >> On Tue, Jan 19, 2016 at 11:12 AM, Maxim Khutornenko <
> maxim@apache.org>
> >> >> wrote:
> >> >>
> >> >> > "Of course, the scaler could manually health check that all
> instances
> >> >> > have come up and are being used as expected, but I guess that
is
> what
> >> >> > Aurora is for."
> >> >> >
> >> >> > I'd argue the updater "watch_secs" health checking isn't enough
to
> >> >> > ensure graceful rollout as instances may start flapping right
after
> >> >> > the updater signs off. Instances outside of update window may
also
> >> >> > flap (e.g. due to backend pressure) and updater will not be able
to
> >> >> > catch that. That's why a robust autoscaler has to rely on external
> >> >> > monitoring tools and overall job health instead.
> >> >> >
> >> >> > A very basic approach, as you mentioned above, could be querying
> job
> >> >> > status repeatedly and count the ratio of tasks in RUNNING vs active
> >> >> > (ASSIGNED, PENDING, THROTTLED, STARTING, etc.) states in order
to
> make
> >> >> > a scaleOut decision. The more reliable approach though would also
> rely
> >> >> > on external monitoring stats exposed by user processes. That would
> be
> >> >> > a much higher fidelity signal than a decision based on task status
> >> >> > alone. Scheduler does not (and should not for scalability reasons)
> >> >> > have visibility into those stats, so the autoscaler would be in
a
> much
> >> >> > better position to make an executive decision there.
> >> >> >
> >> >> > On Sun, Jan 17, 2016 at 9:00 AM, Erb, Stephan
> >> >> > <Stephan.Erb@blue-yonder.com> wrote:
> >> >> > > I believe the operation is not that simple when you look
at the
> >> >> > end-to-end scenario.
> >> >> > >
> >> >> > > For example, the implementation of an auto-scaler  using
the new
> >> >> > scaleOut() API could look like:
> >> >> > >
> >> >> > > 1) check some KPI
> >> >> > > 2) Infer an action based on this KPI such as scaleUp() or
> >> scaleDown()
> >> >> > > 3) wait until the effects of the adjusted instance count
is
> >> reflected
> >> >> in
> >> >> > the KPI. Go to  1 and repeat.
> >> >> > >
> >> >> > > The health checking capabilities of the existing updater
(in
> >> particular
> >> >> > together with [1]) would be really helpful here. Still, the
> simplified
> >> >> > scaleOut() API would offer the great benefit that the auto-scaler
> >> would
> >> >> not
> >> >> > need to know about the used aurora configuration.
> >> >> > >
> >> >> > > We even had an incident with a sub-optimal implementation
of step
> >> 3):
> >> >> An
> >> >> > overloaded package backend lead to slow service startups. The
> service
> >> >> > startup took longer than the grace-period of our auto-scaler.
It
> >> >> therefore
> >> >> > decided to add more and more instances, because the KPI wasn't
> >> improving
> >> >> as
> >> >> > expected. It had no way of knowing that these instances were not
> even
> >> >> > 'running'. The additionally added instances aggravated the overload
> >> >> > situation of the package backend.  Of course, the scaler could
> >> manually
> >> >> > health check that all instances have come up and are being used
as
> >> >> > expected, but I guess that is what Aurora is for.
> >> >> > >
> >> >> > > [1]
> >> >> >
> >> >>
> >>
> https://docs.google.com/document/d/1ZdgW8S4xMhvKW7iQUX99xZm10NXSxEWR0a-21FP5d94/edit?pref=2&pli=1#heading=h.n0kb37aiy8ua
> >> >> > >
> >> >> > > Best Regards,
> >> >> > > Stephan
> >> >> > > ________________________________________
> >> >> > > From: Maxim Khutornenko <maxim@apache.org>
> >> >> > > Sent: Friday, January 15, 2016 7:06 PM
> >> >> > > To: dev@aurora.apache.org
> >> >> > > Subject: Re: [PROPOSAL] Job instance scaling APIs
> >> >> > >
> >> >> > > I wasn't planning on using the rolling updater functionality
> given
> >> the
> >> >> > > simplicity of the operation. I'd second Steve's earlier concerns
> >> about
> >> >> > > scaleOut() looking more like startJobUpdate() if we keep
adding
> >> >> > > features. If health watching, throttling (batch_size) or
> rollback on
> >> >> > > failure is required then I believe the startJobUpdate() should
be
> >> used
> >> >> > > instead of scaleOut().
> >> >> > >
> >> >> > > On Fri, Jan 15, 2016 at 1:09 AM, Erb, Stephan
> >> >> > > <Stephan.Erb@blue-yonder.com> wrote:
> >> >> > >> I really like the proposal. The gain in simplicity on
the
> >> client-side
> >> >> > by not having to provide an aurora config is quite significant.
> >> >> > >>
> >> >> > >> The implementation on the scheduler side is probably
rather
> >> straight
> >> >> > forward as the update can be reused. That would also provide us
> with
> >> the
> >> >> > update UI, which has shown to be quite useful when tracing
> autoscaler
> >> >> > events.
> >> >> > >>
> >> >> > >> Regards,
> >> >> > >> Stephan
> >> >> > >> ________________________________________
> >> >> > >> From: Maxim Khutornenko <maxim@apache.org>
> >> >> > >> Sent: Thursday, January 14, 2016 9:50 PM
> >> >> > >> To: dev@aurora.apache.org
> >> >> > >> Subject: Re: [PROPOSAL] Job instance scaling APIs
> >> >> > >>
> >> >> > >> "I'd be concerned that any
> >> >> > >> scaling API to be powerful enough to fit all (most) use
cases
> would
> >> >> just
> >> >> > >> end up looking like the update API."
> >> >> > >>
> >> >> > >> There is a big difference between scaleOut and startJobUpdate
> APIs
> >> >> > >> that justifies the inclusion of the former. Namely, scaleOut
may
> >> only
> >> >> > >> replicate the existing instances without changing/introducing
> any
> >> new
> >> >> > >> scheduling requirements or performing instance
> rollout/rollback. I
> >> >> > >> don't see scaleOut ever becoming more powerful to threaten
> >> >> > >> startJobUpdate. At the same time, the absence of aurora
config
> >> >> > >> requirement is a huge boost to autoscaling client
> simplification.
> >> >> > >>
> >> >> > >> "For example, when scaling down we don't just kill the
last N
> >> >> > instances, we
> >> >> > >> actually look at the least loaded hosts (globally) and
kill
> tasks
> >> from
> >> >> > >> those."
> >> >> > >>
> >> >> > >> I don't quite see why the same wouldn't be possible with
a
> scaleIn
> >> >> > >> API. Isn't it always external process responsibility
to pay due
> >> >> > >> diligence before killing instances?
> >> >> > >>
> >> >> > >>
> >> >> > >> On Thu, Jan 14, 2016 at 12:35 PM, Steve Niemitz <
> >> sniemitz@apache.org>
> >> >> > wrote:
> >> >> > >>> As some background, we handle scale up / down purely
from the
> >> client
> >> >> > side,
> >> >> > >>> using the update API for both directions.  I'd be
concerned
> that
> >> any
> >> >> > >>> scaling API to be powerful enough to fit all (most)
use cases
> >> would
> >> >> > just
> >> >> > >>> end up looking like the update API.
> >> >> > >>>
> >> >> > >>> For example, when scaling down we don't just kill
the last N
> >> >> > instances, we
> >> >> > >>> actually look at the least loaded hosts (globally)
and kill
> tasks
> >> >> from
> >> >> > >>> those.
> >> >> > >>>
> >> >> > >>>
> >> >> > >>> On Thu, Jan 14, 2016 at 3:28 PM, Maxim Khutornenko
<
> >> maxim@apache.org
> >> >> >
> >> >> > wrote:
> >> >> > >>>
> >> >> > >>>> "How is scaling down different from killing instances?"
> >> >> > >>>>
> >> >> > >>>> I found 'killTasks' syntax too different and
way much more
> >> powerful
> >> >> to
> >> >> > >>>> be used for scaling in. The TaskQuery allows
killing instances
> >> >> across
> >> >> > >>>> jobs/roles, whereas 'scaleIn' is narrowed down
to just a
> single
> >> job.
> >> >> > >>>> Additional benefit: it can be ACLed independently
by allowing
> >> >> external
> >> >> > >>>> process kill tasks only within a given job. We
may also add
> rate
> >> >> > >>>> limiting or backoff to it later.
> >> >> > >>>>
> >> >> > >>>> As for Joshua's question, I feel it should be
an operator's
> >> >> > >>>> responsibility to diff a job with its aurora
config before
> >> applying
> >> >> an
> >> >> > >>>> update. That said, if there is enough demand
we can definitely
> >> >> > >>>> consider adding something similar to what George
suggested or
> >> >> > >>>> resurrecting a 'large change' warning message
we used to have
> in
> >> >> > >>>> client updater.
> >> >> > >>>>
> >> >> > >>>> On Thu, Jan 14, 2016 at 12:06 PM, George Sirois
<
> >> >> george@tellapart.com
> >> >> > >
> >> >> > >>>> wrote:
> >> >> > >>>> > As a point of reference, we solved this
problem by adding a
> >> >> binding
> >> >> > >>>> helper
> >> >> > >>>> > that queries the scheduler for the current
number of
> instances
> >> and
> >> >> > uses
> >> >> > >>>> > that number instead of a hardcoded config:
> >> >> > >>>> >
> >> >> > >>>> >    instances='{{scaling_instances[60]}}'
> >> >> > >>>> >
> >> >> > >>>> > In this example, instances will be set to
the currently
> running
> >> >> > number
> >> >> > >>>> > (unless there are none, in which case 60
instances will be
> >> >> created).
> >> >> > >>>> >
> >> >> > >>>> > On Thu, Jan 14, 2016 at 2:44 PM, Joshua
Cohen <
> >> jcohen@apache.org>
> >> >> > wrote:
> >> >> > >>>> >
> >> >> > >>>> >> What happens if a job has been scaled
out, but the
> underlying
> >> >> > config is
> >> >> > >>>> not
> >> >> > >>>> >> updated to take that scaling into account?
Would the next
> >> update
> >> >> > on that
> >> >> > >>>> >> job revert the number of instances (presumably,
because
> what
> >> else
> >> >> > could
> >> >> > >>>> we
> >> >> > >>>> >> do)? Is there anything we can do, tooling-wise,
to improve
> >> upon
> >> >> > this?
> >> >> > >>>> >>
> >> >> > >>>> >> On Thu, Jan 14, 2016 at 1:40 PM, Maxim
Khutornenko <
> >> >> > maxim@apache.org>
> >> >> > >>>> >> wrote:
> >> >> > >>>> >>
> >> >> > >>>> >> > Our rolling update APIs can be
quite inconvenient to work
> >> with
> >> >> > when it
> >> >> > >>>> >> > comes to instance scaling [1].
It's especially
> frustrating
> >> when
> >> >> > >>>> >> > adding/removing instances has to
be done in an automated
> >> >> fashion
> >> >> > >>>> (e.g.:
> >> >> > >>>> >> by
> >> >> > >>>> >> > an external autoscaling process)
as it requires holding
> on
> >> to
> >> >> the
> >> >> > >>>> >> original
> >> >> > >>>> >> > aurora config at all times.
> >> >> > >>>> >> >
> >> >> > >>>> >> > I propose we add simple instance
scaling APIs to address
> the
> >> >> > above.
> >> >> > >>>> Since
> >> >> > >>>> >> > Aurora job may have instances at
different configs at any
> >> >> > moment, I
> >> >> > >>>> >> propose
> >> >> > >>>> >> > we accept an InstanceKey as a reference
point when
> scaling
> >> out.
> >> >> > For
> >> >> > >>>> >> > example:
> >> >> > >>>> >> >
> >> >> > >>>> >> >     /** Scales out a given job
by adding more instances
> with
> >> >> the
> >> >> > task
> >> >> > >>>> >> > config of the templateKey. */
> >> >> > >>>> >> >     Response scaleOut(1: InstanceKey
templateKey, 2: i32
> >> >> > >>>> incrementCount)
> >> >> > >>>> >> >
> >> >> > >>>> >> >     /** Scales in a given job by
removing existing
> >> instances.
> >> >> */
> >> >> > >>>> >> >     Response scaleIn(1: JobKey
job, 2: i32
> decrementCount)
> >> >> > >>>> >> >
> >> >> > >>>> >> > A correspondent client command
could then look like:
> >> >> > >>>> >> >
> >> >> > >>>> >> >     aurora job scale-out devcluster/vagrant/test/hello/1
> 10
> >> >> > >>>> >> >
> >> >> > >>>> >> > For the above command, a scheduler
would take task
> config of
> >> >> > instance
> >> >> > >>>> 1
> >> >> > >>>> >> of
> >> >> > >>>> >> > the 'hello' job and replicate it
10 more times thus
> adding
> >> 10
> >> >> > >>>> additional
> >> >> > >>>> >> > instances to the job.
> >> >> > >>>> >> >
> >> >> > >>>> >> > There are, of course, some details
to work out like
> making
> >> sure
> >> >> > no
> >> >> > >>>> active
> >> >> > >>>> >> > update is in flight, scale out
does not violate quota and
> >> etc.
> >> >> I
> >> >> > >>>> intend
> >> >> > >>>> >> to
> >> >> > >>>> >> > address those during the implementation
as things
> progress.
> >> >> > >>>> >> >
> >> >> > >>>> >> > Does the above make sense? Any
concerns/suggestions?
> >> >> > >>>> >> >
> >> >> > >>>> >> > Thanks,
> >> >> > >>>> >> > Maxim
> >> >> > >>>> >> >
> >> >> > >>>> >> > [1] - https://issues.apache.org/jira/browse/AURORA-1258
> >> >> > >>>> >> >
> >> >> > >>>> >>
> >> >> > >>>>
> >> >> >
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message