Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EE7F0FF75 for ; Fri, 19 Apr 2013 15:43:53 +0000 (UTC) Received: (qmail 63021 invoked by uid 500); 19 Apr 2013 15:43:51 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 62898 invoked by uid 500); 19 Apr 2013 15:43:50 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 62857 invoked by uid 99); 19 Apr 2013 15:43:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Apr 2013 15:43:49 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: unknown (nike.apache.org: error in processing during lookup of kais@neteck-fr.com) Received: from [209.85.219.48] (HELO mail-oa0-f48.google.com) (209.85.219.48) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Apr 2013 15:43:43 +0000 Received: by mail-oa0-f48.google.com with SMTP id f4so2219183oah.35 for ; Fri, 19 Apr 2013 08:43:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type:x-gm-message-state; bh=ESTJBNudqo6lwGV25ab7FnWWCcm/UFa0yDbtHeY089o=; b=BciNe/6vn76Qfsuv8/ODi2ZTkiE52d4pjw2whxGuOP+wAQfMIJqgmo+T2XVEKglQj6 LOJ1Ccuq3F1kxDBXcovwBQ4TPF8aLog1OhwlscFRk0SneG+/S+9MjJVBi2gKgANKQ+bE v7H6p0s7dnCwZcWGnkDnFFR0GB9xvozSoocf5U3aD0TjLUiTsjpB73Kq/YY3309tKgLg sZp3JHfW5Y4djKL7ZLJgCLCkG4JCd0dJOxtmsfng1qGHJpCizVFad8sYzpVvqj+5ECXA Ny+0UI4Q9Mp2+dZtwYksKXt2xHykYXyZZ6aPFyBsWJk1cre4XGyIzlqDkAS8LBcfuAFO EfFA== MIME-Version: 1.0 X-Received: by 10.182.89.193 with SMTP id bq1mr4207184obb.41.1366386196576; Fri, 19 Apr 2013 08:43:16 -0700 (PDT) Received: by 10.182.80.71 with HTTP; Fri, 19 Apr 2013 08:43:16 -0700 (PDT) In-Reply-To: <02DD1960-A0E4-49B1-B821-6B98807CFB27@thelastpickle.com> References: <02DD1960-A0E4-49B1-B821-6B98807CFB27@thelastpickle.com> Date: Fri, 19 Apr 2013 17:43:16 +0200 Message-ID: Subject: Re: Moving cluster From: Kais Ahmed To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e013cc396dfdf3a04dab893c9 X-Gm-Message-State: ALoCoQlCZCQa2d8YykSGbIkpJjzHJUTQCmv672krppXX4JHw/zNdUh/NQa0y1UvHcj218G/H+Vgf X-Virus-Checked: Checked by ClamAV on apache.org --089e013cc396dfdf3a04dab893c9 Content-Type: text/plain; charset=ISO-8859-1 Hello and thank you for your answers. The first solution is much easier for me because I use the vnode. What is the risk of the first solution thank you, 2013/4/18 aaron morton > This is roughly the lift and shift process I use. > > Note that disabling thrift and gossip does not stop an existing repair > session. So I often drain and then shutdown, and copy the live data dir > rather than a snapshot dir. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 19/04/2013, at 4:10 AM, Michael Theroux wrote: > > This should work. > > Another option is to follow a process similar to what we recently did. We > recently and successfully upgraded 12 instances from large to xlarge > instances in AWS. I chose not to replace nodes as restoring data from the > ring would have taken significant time and put the cluster under some > additional load. I also wanted to eliminate the possibility that any > issues on the new nodes could be blamed on new configuration/operating > system differences. Instead we followed the following procedure (removing > some details that would likely be unique to our infrastructure). > > For a node being upgraded: > > 1) nodetool disable thrift > 2) nodetool disable gossip > 3) Snapshot the data (nodetool snapshot ...) > 4) Backup the snapshot data to EBS (assuming you are on ephemeral) > 5) Stop cassandra > 6) Move the cassandra.yaml configuration file to cassandra.yaml.bak (to > prevent any future restarts to cause cassandra to restart) > 7) Shutdown the instance > 8) Take an AMI of the instance > 9) Start a new instance from the AMI with the desired hardware > 10) If you assign the new instance a new IP Address, make sure any entries > in /etc/hosts, or the broadcast_address in cassandra.yaml is updated > 11) Attach the volume you backed up your snapshot data to to the new > instance and mount it > 12) Restore the snapshot data > 13) Restore cassandra.yaml file > 13) Restart cassandra > > - I recommend practicing this on a test cluster first > - As you replace nodes with new IP Addresses, eventually all your seeds > will need be updated. This is not a big deal until all your seed nodes > have been replaced. > - Don't forget about NTP! Make sure it is running on all your new nodes. > Myself, to be extra careful, I actually deleted the ntp drift file and let > NTP recalculate it because its a new instance, and it took over an hour to > restore our snapshot data... but that may have been overkill. > - If you have the opportunity, depending on your situation, increase > the max_hint_window_in_ms > - Your details may vary > > Thanks, > -Mike > > On Apr 18, 2013, at 11:07 AM, Alain RODRIGUEZ wrote: > > I would say add your 3 servers to the 3 tokens where you want them, let's > say : > > { > "0": { > "0": 0, > "1": 56713727820156410577229101238628035242, > "2": 113427455640312821154458202477256070485 > } > } > > or these token -1 or +1 if you already have these token used. And then > just decommission x1Large nodes. You should be good to go. > > > > 2013/4/18 Kais Ahmed > >> Hi, >> >> What is the best pratice to move from a cluster of 7 nodes (m1.xlarge) to >> 3 nodes (hi1.4xlarge). >> >> Thanks, >> > > > > --089e013cc396dfdf3a04dab893c9 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hello and thank you f= or your answers.

The first solutio= n is much easier for me bec= ause I use the vnode= .

What is the risk of the= first solution

thank you,


2013/4/18 aaron morton <aaron@the= lastpickle.com>
This is roughly the lift and shift proc= ess I use.=A0

Note that disabling thrift and gossip does= not stop an existing repair session. So I often drain and then shutdown, a= nd copy the live data dir rather than a snapshot dir.=A0

Cheers
=A0
-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 19/04/2013, at 4:10 AM, Michael Theroux <mtheroux2@yahoo.com> wrot= e:

T= his should work. =A0

Another option is to follow a process similar to what we rec= ently did. =A0We recently and successfully upgraded 12 instances from large= to xlarge instances in AWS. =A0I chose not to replace nodes as restoring d= ata from the ring would have taken significant time and put the cluster und= er some additional load. =A0I also wanted to eliminate the possibility that= any issues on the new nodes could be blamed on new configuration/operating= system differences. =A0Instead we followed the following procedure (removi= ng some details that would likely be unique to our infrastructure).

For a node being upgraded:

1) nodet= ool disable thrift=A0
2) nodetool disable gossip
3) Sna= pshot the data (nodetool snapshot ...)
4) Backup the snapshot dat= a to EBS (assuming you are on ephemeral)
5) Stop cassandra
6) Move the cassandra.yaml configuration f= ile to cassandra.yaml.bak (to prevent any future restarts to cause cassandr= a to restart)
7) Shutdown the instance
8) Take an AMI o= f the instance
9) Start a new instance from the AMI with the desired hardware
10) If you assign the new instance a new IP Address, make sure any entri= es in /etc/hosts, or the broadcast_address in cassandra.yaml is updated
11) Attach the volume you backed up your snapshot data to to the new i= nstance and mount it
12) Restore the snapshot data
13) = Restore cassandra.yaml file
13) Restart cassandra

- I recommend practicing this on a test cluster first
= - As you replace nodes with new IP Addresses, eventually all your seeds wil= l need be updated. =A0This is not a big deal until all your seed nodes have= been replaced.
- Don't forget about NTP! =A0Make sure it is running on all your n= ew nodes. =A0Myself, to be extra careful, I actually deleted the ntp drift = file and let NTP recalculate it because its a new instance, and it took ove= r an hour to restore our snapshot data... but that may have been overkill.<= /div>
- If you have the opportunity, depending on your situation, increase t= he=A0max_hint_window_in_ms
- Your details may vary

=
Thanks,
-Mike

On Apr 18, 2013,= at 11:07 AM, Alain RODRIGUEZ wrote:

I would say add your 3 serve= rs to the 3 tokens where you want them, let's say :

=
{
=A0 =A0 "0": {
=A0 =A0 =A0 =A0 "0= ": 0,
=A0 =A0 =A0 =A0 "1": 56713727820156410577229101238628035242,=
=A0 =A0 =A0 =A0 "2": 113427455640312821154458202477256070485=
=A0 =A0 }
}

or these to= ken -1 or +1 if you already have these token used. And then just decommissi= on x1Large nodes. You should be good to go.



2013/4/18 Kais Ahmed <kais@neteck-fr.com>
Hi,

What is the best pratice to mov= e from a cluster of 7 nodes (m1.xlarge) to 3 nodes (hi1.4xlarge).

Thanks,




--089e013cc396dfdf3a04dab893c9--