Hello,

We have Cassandra cluster with 5 nodes hosted in the Amazon EC2, and  I had to restart two of them, so their IPs changed.
We use NetworkTopologyStrategy, so I simply updated IPs in the cassandra-topology.properties file.

However, as I understood, old IPs remained somewhere in the system keyspace, and now I observe several different exception stacktraces in the log files, including:

java.lang.RuntimeException: Host ID collision between active endpoint /<new IP> and /<old IP> (id=ab66dd02-96b2-4504-8403-7d066f911698)
        at org.apache.cassandra.locator.TokenMetadata.updateHostId(TokenMetadata.java:229)
        at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1358)
        at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1228)
        at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1960)
        at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:837)
        at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:915)
        at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:50)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

and

java.lang.AssertionError: Missing host ID for <old IP>
        at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:583)
        at org.apache.cassandra.service.StorageProxy$5.runMayThrow(StorageProxy.java:552)
        at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1658)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)


nodetool status being executed on 3 old nodes, shows old ghost node:

Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns   Host ID                               Rack
UN  10.14.128.109  2.8 GB     141     4.1%   32260392-12c2-4f1a-812e-87fd9a960d10  RAC2
UN  10.24.33.187   2.12 GB    258     42.7%  ab66dd02-96b2-4504-8403-7d066f911698  RAC3
UN  10.20.149.165  2.99 GB    251     4.5%   a0792f59-20b1-4017-a7f6-88e0c0d7f86f  RAC1
DN  10.11.73.104   1.07 GB    2       1.0%   null                                  RAC1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns   Host ID                               Rack
UN  10.34.78.23     2.21 GB    117     0.9%   2acd3766-404d-4cdc-b3e3-7b3b95856f0e  RAC1
UN  10.20.23.171   2.22 GB    255     46.8%  67421e3a-1dfc-48a0-88b3-c6dbd64dc9d8  RAC1


Is it possible to fix those IP collisions ?


Thanks.