aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Brazil <brian.bra...@boxever.com>
Subject Re: [jira] [Created] (AURORA-1258) Improve procedure for adding instances to a job
Date Wed, 08 Apr 2015 08:16:12 GMT
On 7 April 2015 at 20:45, Kevin Sweeney <ksweeney@twitter.com.invalid>
wrote:

> How about a scale command somewhere that will check that the only operation
> it's doing is "scaling" and not introducing new code. This could be
> accomplished by diffing the config for the new instances with that of the
> old instances. For cases where it's not a pure scale operation, maybe have
> an override flag like --allow-heterogenous. Also, the existing updater
> should handle this case fine if the config hasn't changed - it won't
> attempt to modify unchanged instances. Maybe we need to update diff to give
> an easy way to ensure that a proposed change is a pure scale operation and
> not a combination of scaling and a code deploy.
>

There are other types of change that it'd be good to do without changing
code/flags, such as giving a job more resources if it's OOMing.

Some size changes may also require changing flags (e.g. if there's sharding
of some form and each task needs to know the size of the job), so I think
Joe's idea of adding new instances first when the size of the job in
increasing is good as it makes it harder to accidentally take out the
entire job when upsizing.

Brian


>
> On Tuesday, April 7, 2015, Joe Smith (JIRA) <jira@apache.org> wrote:
>
> > Joe Smith created AURORA-1258:
> > ---------------------------------
> >
> >              Summary: Improve procedure for adding instances to a job
> >                  Key: AURORA-1258
> >                  URL: https://issues.apache.org/jira/browse/AURORA-1258
> >              Project: Aurora
> >           Issue Type: Story
> >           Components: Reliability, Usability
> >             Reporter: Joe Smith
> >
> >
> > The current process for adding instances to a job is highly manual, and
> > potentially dangerous.
> >
> > 1. Take a config for a job with 10 instances, update it to 20 instances.
> > 2. The batch size will be increased, and users will need to specify
> shards
> > 10 to 19.
> > 3. After this update is complete, users will need to manually update
> > shards 0-9 again.
> >
> > There may be other changes pulled in as part of this update other than
> > just increasing the number of instances, which could further complicate
> > things.
> >
> > One possible improvement would be to change the updater from
> > 'under-provision' where it kills instances first, then schedules new
> > instances, to an 'over-provision' where it adds on new instances, then
> > backpedals and kills the old instances.
> >
> > Overall, a single command or process for a user to take an
> > already-existing job and increase the number of instances would reduce
> > overhead and fat-fingering.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>
>
> --
> Sent from Gmail Mobile
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message