cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henrik Schröder <>
Subject Re: Moving existing nodes to a different network.
Date Sun, 11 Dec 2011 12:17:52 GMT
I'm running Cassandra 1.0.1 if that makes any difference.


On Sun, Dec 11, 2011 at 13:16, Henrik Schröder <> wrote:

> I have an existing cluster of four Cassandra nodes. The machines have both
> an internal and an external IP, and originally I set them up to use the
> external network. A little while later I moved them to the internal network
> by bringing all machines down, changing the config, and bringing them up
> again. In the logs I found they all said "Changing ownership of token XXX",
> and nodetool ring reported that the cluster consisted of those four
> machines on their internal ips. After that, as part of a cleanup process, I
> moved the tokens on all machines to make sure the cluster was balanced, and
> it also worked perfectly.
> However, now I have to temporarily move the cluster back to the external
> network for a little while. I tried doing the same thing as last time,
> bringing all nodes down, changing the config (rpc address, gossip address,
> list of seeds) and bringing them up again, but this resulted in a very
> confused cluster. When I ran nodetool ring, it reported eight nodes, the
> four internal ips were marked as down, and the four external were marked as
> up, but with the token they had when they previously used that ip. Checking
> the logs, there was no token ownership change, all nodes picked the saved
> token they had when they last used the external ip, and not the token they
> should have, the one I moved each server to when on the internal ip.
> I immediately moved all servers back to the internal IP, and then nodetool
> reported the same as before, a cluster of four machines, all up, and all on
> the token they're supposed to have. No mention of the external ips or the
> old tokens they had there.
> How do I reset this data? Where is it stored? Why does it store all of
> this when nodetool doesn't report it? Why does a node store several saved
> tokens? How do I change their ip without losing any data and without having
> to do removetoken or similar?
> One thought I have is to bring down one node, delete the system keyspace,
> and bring it back up, at which point it would only use what's in the
> config, but fetch the schema from the other nodes. Or would it also fetch
> the old information of what token it had when it was on the external ip? Or
> would something else go wrong?
> /Henrik

View raw message