Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of sdolgy@gmail.com designates
 209.85.220.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=pv3LFvNvvVcSDKsp7ybq3T2gxxbwqb5dy58+UzWFZZcQGaxlWFagKERUR2GPITd6YL
         GZvDwe8m60NYVZdyD0ojsloDyZP/ph1xaFxODqch0Aeqf9VvkSy/PdnNh5P4wgmwH2VL
         +AW17KoWkgpB5fo18BhSorAt4XsM7JnsRPvwU=
MIME-Version: 1.0
In-Reply-To: <7BADE095-DEF5-4F6D-A5D3-A2BBAB362844@synchromedia.co.uk>
References: <6AF0D2D6-EB8B-47BC-800C-F9AD87B21E0B@synchromedia.co.uk>
 <BANLkTinT7dZurE5Mds=18+iZ_1aiY2xQRg@mail.gmail.com>
 <A516BB5C-71E1-46AB-B900-AE143F89629F@synchromedia.co.uk>
 <BANLkTin6xOGU7t_wXQh_y_hjb9ywbnn2Ng@mail.gmail.com>
 <7BADE095-DEF5-4F6D-A5D3-A2BBAB362844@synchromedia.co.uk>
From: Sasha Dolgy <sdolgy@gmail.com>
Date: Thu, 26 May 2011 15:21:32 +0200
Message-ID: <BANLkTi=_b6FTNcEC8UmUddCJZ_KgpJ9RRg@mail.gmail.com>
Subject: Re: EC2 node adding trouble
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, May 26, 2011 at 3:12 PM, Marcus Bointon
<marcus@synchromedia.co.uk> wrote:
> I'd like to make sure I've got the right sequence of operations for addin=
g a node without downtime. If I'm going from 2 to 3 nodes:
>
> 1 Calculate new initial_token values using the python script
> 2 Change token values in existing nodes and restart them
> 3 Install/configure new node
> 4 Insert new node's token value
> 5 Set new node to auto-bootstrap
> 6 Start cassandra on new node
> 7 Wait for the ring to rebalance
>
> With token changes (using values from the python script), it's clear that=
 all nodes will have some data moved. Does this mean that there's a possibi=
lity of overlap between regions if token changes are not absolutely simulta=
neous on all nodes? That sounds dangerous to me... Or shouldn't token value=
s be changed on nodes containing data?
>

nodetool repair is good.  when we add new nodes, we add a new one
without specifying the new token.  after everything is up and healthy,
we determine new tokens and see if there is a need to renumber nodes.
if we do, we do one at a time and wait until the nodetool repair is
finished on one node before moving to another....

> Is there a corresponding sequence for removing nodes? I'm guessing draini=
ng is involved.

Turn the node off, remove the node from the ring using nodetool and
removetoken .... i've found this to be the best problem-free way.
Maybe it's better now ...
http://blog.sasha.dolgy.com/2011/03/apache-cassandra-nodetool.html