cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henrik Schröder <>
Subject Moving existing nodes to a different network.
Date Sun, 11 Dec 2011 12:16:57 GMT
I have an existing cluster of four Cassandra nodes. The machines have both
an internal and an external IP, and originally I set them up to use the
external network. A little while later I moved them to the internal network
by bringing all machines down, changing the config, and bringing them up
again. In the logs I found they all said "Changing ownership of token XXX",
and nodetool ring reported that the cluster consisted of those four
machines on their internal ips. After that, as part of a cleanup process, I
moved the tokens on all machines to make sure the cluster was balanced, and
it also worked perfectly.

However, now I have to temporarily move the cluster back to the external
network for a little while. I tried doing the same thing as last time,
bringing all nodes down, changing the config (rpc address, gossip address,
list of seeds) and bringing them up again, but this resulted in a very
confused cluster. When I ran nodetool ring, it reported eight nodes, the
four internal ips were marked as down, and the four external were marked as
up, but with the token they had when they previously used that ip. Checking
the logs, there was no token ownership change, all nodes picked the saved
token they had when they last used the external ip, and not the token they
should have, the one I moved each server to when on the internal ip.

I immediately moved all servers back to the internal IP, and then nodetool
reported the same as before, a cluster of four machines, all up, and all on
the token they're supposed to have. No mention of the external ips or the
old tokens they had there.

How do I reset this data? Where is it stored? Why does it store all of this
when nodetool doesn't report it? Why does a node store several saved
tokens? How do I change their ip without losing any data and without having
to do removetoken or similar?

One thought I have is to bring down one node, delete the system keyspace,
and bring it back up, at which point it would only use what's in the
config, but fetch the schema from the other nodes. Or would it also fetch
the old information of what token it had when it was on the external ip? Or
would something else go wrong?


View raw message