Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Sender: Michael Shuler <mshuler@pbandjelly.org>
Subject: Re: How do you do automatic restacking of AWS instance for cassandra?
To: user@cassandra.apache.org
References: <CACYuknZnrv08ASGTPDuBzE8nn9=mrZQdg=E_8683EK8=sW5MOw@mail.gmail.com>
 <CAOUOv0GVLZRECeG_3LCgS3m1BzEmcgJ30ZJsfcVpNJoKs59Qcg@mail.gmail.com>
 <CAFcjB9x0aBS3ZS5k87YdVAXGcNSriK+EXBy5LTGATZcwef4=Yw@mail.gmail.com>
 <CACYuknbJkmguBuUj2CN=uSTuLn+zubuYbETxY6j_=4CewM_8LA@mail.gmail.com>
 <pony-cb367aa5aed1991b197e48bcdd89864c2a894a70-8b1bf06d076ef28bdb5f163465f19e015a911fcf@user.cassandra.apache.org>
 <CACYuknbOK6eEYYReuEY_=mRYhVb-xDUrwJpQiK76nqJAbTG=hg@mail.gmail.com>
From: Michael Shuler <michael@pbandjelly.org>
Message-ID: <f06fa1d1-613a-81f5-217a-e87e231eda69@pbandjelly.org>
Date: Sat, 27 May 2017 21:40:32 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <CACYuknbOK6eEYYReuEY_=mRYhVb-xDUrwJpQiK76nqJAbTG=hg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sun, 28 May 2017 02:40:40 -0000

On 05/27/2017 09:13 PM, Surbhi Gupta wrote:
> We get the new AMI release with the new OS updates and we are not
> allowed to use the old AMI .

Neither of Jeff's suggestions use the old AMI - that was my suggestion :)

> On Sat, May 27, 2017 at 7:11 PM Jeff Jirsa <jjirsa@apache.org
> <mailto:jjirsa@apache.org>> wrote:
> 
>     At a previous employer, they used M4 class instances with data on a
>     dedicated EBS volumes, so we could swap AMIs / stop / start / adjust
>     instances without having to deal with this. This worked reasonably
>     well for their scale (which was petabytes of data).

This suggestion seems like a nice way to easily cycle out old AMIs to a
new AMI quickly, getting rid of any data migration. A one-time pain to
switch to EBS with many benfits, which lends itself to automation well.

>     Other companies using ephemeral tend to be more willing to just
>     terminate instances and replace them (-Dcassandra.replace_address).
>     If you stop cassandra, then boot a replacement with
>     'replace_address' set, it'll take over for the stopped instance,
>     including re-streaming all data (as best it can, subject to
>     consistency level and repair status). This may be easier for you to
>     script than switching your fleet to EBS, but it's not without risk.

This suggestion for ephemeral data disks also seems like a possibly more
manageable way to cycle out old AMIs to a new AMI via individual node
replacement. The rest of the cluster needs to be healthy, and it may
mean an almost continuous node replacement, depending on node count.
This is potentially a lot less overall cluster load, since one
replacement node is streaming at a time, instead of all the data
streaming to a new replacement DC. Easier to automate than a replacement
DC, perhaps, too.

-- 
Kind regards,
Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org