aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Khutornenko <ma...@apache.org>
Subject Re: [PROPOSAL] Job instance scaling APIs
Date Tue, 19 Jan 2016 19:12:11 GMT
"Of course, the scaler could manually health check that all instances
have come up and are being used as expected, but I guess that is what
Aurora is for."

I'd argue the updater "watch_secs" health checking isn't enough to
ensure graceful rollout as instances may start flapping right after
the updater signs off. Instances outside of update window may also
flap (e.g. due to backend pressure) and updater will not be able to
catch that. That's why a robust autoscaler has to rely on external
monitoring tools and overall job health instead.

A very basic approach, as you mentioned above, could be querying job
status repeatedly and count the ratio of tasks in RUNNING vs active
(ASSIGNED, PENDING, THROTTLED, STARTING, etc.) states in order to make
a scaleOut decision. The more reliable approach though would also rely
on external monitoring stats exposed by user processes. That would be
a much higher fidelity signal than a decision based on task status
alone. Scheduler does not (and should not for scalability reasons)
have visibility into those stats, so the autoscaler would be in a much
better position to make an executive decision there.

On Sun, Jan 17, 2016 at 9:00 AM, Erb, Stephan
<Stephan.Erb@blue-yonder.com> wrote:
> I believe the operation is not that simple when you look at the end-to-end scenario.
>
> For example, the implementation of an auto-scaler  using the new scaleOut() API could
look like:
>
> 1) check some KPI
> 2) Infer an action based on this KPI such as scaleUp() or scaleDown()
> 3) wait until the effects of the adjusted instance count is reflected in the KPI. Go
to  1 and repeat.
>
> The health checking capabilities of the existing updater (in particular together with
[1]) would be really helpful here. Still, the simplified scaleOut() API would offer the great
benefit that the auto-scaler would not need to know about the used aurora configuration.
>
> We even had an incident with a sub-optimal implementation of step 3): An overloaded package
backend lead to slow service startups. The service startup took longer than the grace-period
of our auto-scaler. It  therefore decided to add more and more instances, because the KPI
wasn't improving as expected. It had no way of knowing that these instances were not even
'running'. The additionally added instances aggravated the overload situation of the package
backend.  Of course, the scaler could manually health check that all instances have come up
and are being used as expected, but I guess that is what Aurora is for.
>
> [1] https://docs.google.com/document/d/1ZdgW8S4xMhvKW7iQUX99xZm10NXSxEWR0a-21FP5d94/edit?pref=2&pli=1#heading=h.n0kb37aiy8ua
>
> Best Regards,
> Stephan
> ________________________________________
> From: Maxim Khutornenko <maxim@apache.org>
> Sent: Friday, January 15, 2016 7:06 PM
> To: dev@aurora.apache.org
> Subject: Re: [PROPOSAL] Job instance scaling APIs
>
> I wasn't planning on using the rolling updater functionality given the
> simplicity of the operation. I'd second Steve's earlier concerns about
> scaleOut() looking more like startJobUpdate() if we keep adding
> features. If health watching, throttling (batch_size) or rollback on
> failure is required then I believe the startJobUpdate() should be used
> instead of scaleOut().
>
> On Fri, Jan 15, 2016 at 1:09 AM, Erb, Stephan
> <Stephan.Erb@blue-yonder.com> wrote:
>> I really like the proposal. The gain in simplicity on the client-side by not having
to provide an aurora config is quite significant.
>>
>> The implementation on the scheduler side is probably rather straight forward as the
update can be reused. That would also provide us with the update UI, which has shown to be
quite useful when tracing autoscaler events.
>>
>> Regards,
>> Stephan
>> ________________________________________
>> From: Maxim Khutornenko <maxim@apache.org>
>> Sent: Thursday, January 14, 2016 9:50 PM
>> To: dev@aurora.apache.org
>> Subject: Re: [PROPOSAL] Job instance scaling APIs
>>
>> "I'd be concerned that any
>> scaling API to be powerful enough to fit all (most) use cases would just
>> end up looking like the update API."
>>
>> There is a big difference between scaleOut and startJobUpdate APIs
>> that justifies the inclusion of the former. Namely, scaleOut may only
>> replicate the existing instances without changing/introducing any new
>> scheduling requirements or performing instance rollout/rollback. I
>> don't see scaleOut ever becoming more powerful to threaten
>> startJobUpdate. At the same time, the absence of aurora config
>> requirement is a huge boost to autoscaling client simplification.
>>
>> "For example, when scaling down we don't just kill the last N instances, we
>> actually look at the least loaded hosts (globally) and kill tasks from
>> those."
>>
>> I don't quite see why the same wouldn't be possible with a scaleIn
>> API. Isn't it always external process responsibility to pay due
>> diligence before killing instances?
>>
>>
>> On Thu, Jan 14, 2016 at 12:35 PM, Steve Niemitz <sniemitz@apache.org> wrote:
>>> As some background, we handle scale up / down purely from the client side,
>>> using the update API for both directions.  I'd be concerned that any
>>> scaling API to be powerful enough to fit all (most) use cases would just
>>> end up looking like the update API.
>>>
>>> For example, when scaling down we don't just kill the last N instances, we
>>> actually look at the least loaded hosts (globally) and kill tasks from
>>> those.
>>>
>>>
>>> On Thu, Jan 14, 2016 at 3:28 PM, Maxim Khutornenko <maxim@apache.org> wrote:
>>>
>>>> "How is scaling down different from killing instances?"
>>>>
>>>> I found 'killTasks' syntax too different and way much more powerful to
>>>> be used for scaling in. The TaskQuery allows killing instances across
>>>> jobs/roles, whereas 'scaleIn' is narrowed down to just a single job.
>>>> Additional benefit: it can be ACLed independently by allowing external
>>>> process kill tasks only within a given job. We may also add rate
>>>> limiting or backoff to it later.
>>>>
>>>> As for Joshua's question, I feel it should be an operator's
>>>> responsibility to diff a job with its aurora config before applying an
>>>> update. That said, if there is enough demand we can definitely
>>>> consider adding something similar to what George suggested or
>>>> resurrecting a 'large change' warning message we used to have in
>>>> client updater.
>>>>
>>>> On Thu, Jan 14, 2016 at 12:06 PM, George Sirois <george@tellapart.com>
>>>> wrote:
>>>> > As a point of reference, we solved this problem by adding a binding
>>>> helper
>>>> > that queries the scheduler for the current number of instances and uses
>>>> > that number instead of a hardcoded config:
>>>> >
>>>> >    instances='{{scaling_instances[60]}}'
>>>> >
>>>> > In this example, instances will be set to the currently running number
>>>> > (unless there are none, in which case 60 instances will be created).
>>>> >
>>>> > On Thu, Jan 14, 2016 at 2:44 PM, Joshua Cohen <jcohen@apache.org>
wrote:
>>>> >
>>>> >> What happens if a job has been scaled out, but the underlying config
is
>>>> not
>>>> >> updated to take that scaling into account? Would the next update
on that
>>>> >> job revert the number of instances (presumably, because what else
could
>>>> we
>>>> >> do)? Is there anything we can do, tooling-wise, to improve upon
this?
>>>> >>
>>>> >> On Thu, Jan 14, 2016 at 1:40 PM, Maxim Khutornenko <maxim@apache.org>
>>>> >> wrote:
>>>> >>
>>>> >> > Our rolling update APIs can be quite inconvenient to work with
when it
>>>> >> > comes to instance scaling [1]. It's especially frustrating
when
>>>> >> > adding/removing instances has to be done in an automated fashion
>>>> (e.g.:
>>>> >> by
>>>> >> > an external autoscaling process) as it requires holding on
to the
>>>> >> original
>>>> >> > aurora config at all times.
>>>> >> >
>>>> >> > I propose we add simple instance scaling APIs to address the
above.
>>>> Since
>>>> >> > Aurora job may have instances at different configs at any moment,
I
>>>> >> propose
>>>> >> > we accept an InstanceKey as a reference point when scaling
out. For
>>>> >> > example:
>>>> >> >
>>>> >> >     /** Scales out a given job by adding more instances with
the task
>>>> >> > config of the templateKey. */
>>>> >> >     Response scaleOut(1: InstanceKey templateKey, 2: i32
>>>> incrementCount)
>>>> >> >
>>>> >> >     /** Scales in a given job by removing existing instances.
*/
>>>> >> >     Response scaleIn(1: JobKey job, 2: i32 decrementCount)
>>>> >> >
>>>> >> > A correspondent client command could then look like:
>>>> >> >
>>>> >> >     aurora job scale-out devcluster/vagrant/test/hello/1 10
>>>> >> >
>>>> >> > For the above command, a scheduler would take task config of
instance
>>>> 1
>>>> >> of
>>>> >> > the 'hello' job and replicate it 10 more times thus adding
10
>>>> additional
>>>> >> > instances to the job.
>>>> >> >
>>>> >> > There are, of course, some details to work out like making
sure no
>>>> active
>>>> >> > update is in flight, scale out does not violate quota and etc.
I
>>>> intend
>>>> >> to
>>>> >> > address those during the implementation as things progress.
>>>> >> >
>>>> >> > Does the above make sense? Any concerns/suggestions?
>>>> >> >
>>>> >> > Thanks,
>>>> >> > Maxim
>>>> >> >
>>>> >> > [1] - https://issues.apache.org/jira/browse/AURORA-1258
>>>> >> >
>>>> >>
>>>>

Mime
View raw message