cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Olson <>
Subject Changed node ID?
Date Tue, 07 Mar 2017 03:44:50 GMT

I have a 9 node cluster I had shut down (cassandra stopped on all nodes, all nodes shutdown)
that I just tried to start back up. I have done this several times successfully. However,
on this attempt, one of the nodes failed to join the cluster. Upon inspection of /var/log/cassandra/system.log,
I found the following: 

WARN [GossipStage:1] 2017-03-06 21:06:36,648 - Changing /'s
host ID from cff3ef25-9a47-4ea4-9519-b85d20bef3ee to 59f2da9f-0b85-452f-b61a-fa990de53e4b

further down: 

ERROR [main] 2017-03-06 21:20:14,718 - Exception encountered during
java.lang.RuntimeException: A node with address / already exists, cancelling
join. Use cassandra.replace_address if you want to replace this node. 
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(
at org.apache.cassandra.service.StorageService.prepareToJoin( ~[apache-cassandra-3.9.0.jar:3.9.0]

at org.apache.cassandra.service.StorageService.initServer( ~[apache-cassandra-3.9.0.jar:3.9.0]

at org.apache.cassandra.service.StorageService.initServer( ~[apache-cassandra-3.9.0.jar:3.9.0]

at org.apache.cassandra.service.CassandraDaemon.setup( [apache-cassandra-3.9.0.jar:3.9.0]

at org.apache.cassandra.service.CassandraDaemon.activate( [apache-cassandra-3.9.0.jar:3.9.0]

at org.apache.cassandra.service.CassandraDaemon.main( [apache-cassandra-3.9.0.jar:3.9.0]

nodetool status: 

UN 2.58 TiB 256 32.0% 9de2d3ef-5ae1-4c7f-8560-730757a6d1ae rack1 
UN 2.26 TiB 256 33.9% d83829d3-a1d3-4e6c-b014-7cfe45e22d67 rack1 
UN 2.91 TiB 256 34.1% 0cafd24e-d3ed-4e51-b586-0b496835a931 rack1 
DN 551.45 KiB 256 31.9% 59f2da9f-0b85-452f-b61a-fa990de53e4b rack1 
UN 2.32 TiB 256 32.7% db006e31-03fa-486a-8512-f88eb583bd0c rack1 
UN 2.54 TiB 256 34.3% a9a50a74-2fc2-4866-a03a-ec95a7866183 rack1 
UN 2.4 TiB 256 35.9% 733e6703-c18f-432f-a787-3731f80ba42d rack1 
UN 2.34 TiB 256 32.1% 0daa06fa-708f-4ff8-a15e-861f1a53113a rack1 
UN 4.07 TiB 256 33.1% 2aa578c6-1332-4b94-81c6-c3ce005a52ef rack1 

My questions: 
1. Why did the host ID change? 
2. If I modify to include 
JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=", will I recover the data on
the original node? It is still on the node's hard drive.I really don't want to have to restream
2.6TB of data onto a "new" node. 

View raw message