Mailing-List: contact dev-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAFWq12VagU-gc+dt_k3eZiANJTkkMAOkqBKAyEVq6qa0V_4nxQ@mail.gmail.com>
References: 
 <CAOTkfX5XeS4DK-+htSZjULyVA4sSa8UkY=zqWtALyCSNv2uvEg@mail.gmail.com>
	<CAFWq12VagU-gc+dt_k3eZiANJTkkMAOkqBKAyEVq6qa0V_4nxQ@mail.gmail.com>
Date: Fri, 5 Feb 2016 15:07:14 -0800
Message-ID: 
 <CAOTkfX5a96d5+-8cv=2VZtyS_=kHo7_V27MQEoJjWjP9bM7WEw@mail.gmail.com>
Subject: Re: [PROPOSAL] Disallow instance removal in job update
From: Maxim Khutornenko <maxim@apache.org>
To: dev@aurora.apache.org
Content-Type: text/plain; charset=UTF-8

We have had attempts to safeguard client updater command with a
"dangerous change" warning before but it did not get good feedback.
Besides, automated tools/scripts just ignored it.

An alternative could be what George suggest on the scaling API thread
mentioned earlier: automatically bump up instance count to the job
active task count. I'd say this could be an implementation to the
proposal above rather than a safeguard as it accomplishes the exact
same goal.

Bill, do you have any ideas of what that safeguard could be?

On Fri, Feb 5, 2016 at 2:56 PM, Bill Farner <wfarner@apache.org> wrote:
>>
>> the outdated instance count problem will only get worse as automated
>> scaling tools will quickly render existing .aurora config value obsolete
>
>
> This is not a compelling reason to remove functionality.  Sounds like a
> safeguard is needed instead.
>
> On Fri, Feb 5, 2016 at 2:43 PM, Maxim Khutornenko <maxim@apache.org> wrote:
>
>> This is mostly a survey rather than a proposal. How would people think
>> about limiting updater to only adding/updating instances and let
>> killTasks take care of instance removals?
>>
>> We have all heard stories (or happen to create some ourselves) when an
>> outdated instance count value in .aurora config caused unexpected
>> instance removals. Granted, there are plenty of other values in the
>> config that can cause service-wide outage but instance count seems to
>> be the worst in that sense.
>>
>> After the recent refactoring of addInstances and killTasks to act as
>> scaleOut/scaleIn APIs [1], the outdated instance count problem will
>> only get worse as automated scaling tools will quickly render existing
>> .aurora config value obsolete. With that in mind, should we block
>> instance removal in the updater and let an explicit killTasks call be
>> the only acceptable action to reduce instance count? Is there any
>> value (aside from arguable convenience factor) in having
>> startJobUpdate ever killing instances?
>>
>> Thanks,
>> Maxim
>>
>> [1] - http://markmail.org/message/2smaej5n5e54li3g
>>