Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of skrolle@gmail.com designates
 209.85.213.172 as permitted sender)
MIME-Version: 1.0
Date: Sun, 11 Dec 2011 13:16:57 +0100
Message-ID: 
 <CAN3fqkwn7WMFyYDi=sMV758AG_eFU7wSE7wN76XVBTSH3RrRhw@mail.gmail.com>
Subject: Moving existing nodes to a different network.
From: =?ISO-8859-1?Q?Henrik_Schr=F6der?= <skrolle@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=20cf30549a2b9b78ec04b3cffe97

--20cf30549a2b9b78ec04b3cffe97
Content-Type: text/plain; charset=ISO-8859-1

I have an existing cluster of four Cassandra nodes. The machines have both
an internal and an external IP, and originally I set them up to use the
external network. A little while later I moved them to the internal network
by bringing all machines down, changing the config, and bringing them up
again. In the logs I found they all said "Changing ownership of token XXX",
and nodetool ring reported that the cluster consisted of those four
machines on their internal ips. After that, as part of a cleanup process, I
moved the tokens on all machines to make sure the cluster was balanced, and
it also worked perfectly.

However, now I have to temporarily move the cluster back to the external
network for a little while. I tried doing the same thing as last time,
bringing all nodes down, changing the config (rpc address, gossip address,
list of seeds) and bringing them up again, but this resulted in a very
confused cluster. When I ran nodetool ring, it reported eight nodes, the
four internal ips were marked as down, and the four external were marked as
up, but with the token they had when they previously used that ip. Checking
the logs, there was no token ownership change, all nodes picked the saved
token they had when they last used the external ip, and not the token they
should have, the one I moved each server to when on the internal ip.

I immediately moved all servers back to the internal IP, and then nodetool
reported the same as before, a cluster of four machines, all up, and all on
the token they're supposed to have. No mention of the external ips or the
old tokens they had there.

How do I reset this data? Where is it stored? Why does it store all of this
when nodetool doesn't report it? Why does a node store several saved
tokens? How do I change their ip without losing any data and without having
to do removetoken or similar?

One thought I have is to bring down one node, delete the system keyspace,
and bring it back up, at which point it would only use what's in the
config, but fetch the schema from the other nodes. Or would it also fetch
the old information of what token it had when it was on the external ip? Or
would something else go wrong?


/Henrik

--20cf30549a2b9b78ec04b3cffe97
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I have an existing cluster of four Cassandra nodes. The machines have both =
an internal and an external IP, and originally I set them up to use the ext=
ernal network. A little while later I moved them to the internal network by=
 bringing all machines down, changing the config, and bringing them up agai=
n. In the logs I found they all said &quot;Changing ownership of token XXX&=
quot;, and nodetool ring reported that the cluster consisted of those four =
machines on their internal ips. After that, as part of a cleanup process, I=
 moved the tokens on all machines to make sure the cluster was balanced, an=
d it also worked perfectly.<br>
<br>However, now I have to temporarily move the cluster back to the externa=
l network for a little while. I tried doing the same thing as last time, br=
inging all nodes down, changing the config (rpc address, gossip address, li=
st of seeds) and bringing them up again, but this resulted in a very confus=
ed cluster. When I ran nodetool ring, it reported eight nodes, the four int=
ernal ips were marked as down, and the four external were marked as up, but=
 with the token they had when they previously used that ip. Checking the lo=
gs, there was no token ownership change, all nodes picked the saved token t=
hey had when they last used the external ip, and not the token they should =
have, the one I moved each server to when on the internal ip.<br>
<br>I immediately moved all servers back to the internal IP, and then nodet=
ool reported the same as before, a cluster of four machines, all up, and al=
l on the token they&#39;re supposed to have. No mention of the external ips=
 or the old tokens they had there.<br>
<br>How do I reset this data? Where is it stored? Why does it store all of =
this when nodetool doesn&#39;t report it? Why does a node store several sav=
ed tokens? How do I change their ip without losing any data and without hav=
ing to do removetoken or similar?<br>
<br>One thought I have is to bring down one node, delete the system keyspac=
e, and bring it back up, at which point it would only use what&#39;s in the=
 config, but fetch the schema from the other nodes. Or would it also fetch =
the old information of what token it had when it was on the external ip? Or=
 would something else go wrong?<br>
<br><br>/Henrik<br>

--20cf30549a2b9b78ec04b3cffe97--