incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renat Gilfanov <gren...@mail.ru>
Subject How to fix host ID collision?
Date Tue, 03 Sep 2013 21:01:55 GMT

Hello,

We have Cassandra cluster with 5 nodes hosted in the Amazon EC2, and  I had to restart two
of them, so their IPs changed.
We use NetworkTopologyStrategy, so I simply updated IPs in the cassandra-topology.properties
file.

However, as I understood, old IPs remained somewhere in the system keyspace, and now I observe
several different exception stacktraces in the log files, including:

java.lang.RuntimeException: Host ID collision between active endpoint /<new IP> and
/<old IP> (id=ab66dd02-96b2-4504-8403-7d066f911698)
        at org.apache.cassandra.locator.TokenMetadata.updateHostId(TokenMetadata.java:229)
        at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1358)
        at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1228)
        at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1960)
        at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:837)
        at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:915)
        at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:50)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

and

java.lang.AssertionError: Missing host ID for <old IP>
        at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:583)
        at org.apache.cassandra.service.StorageProxy$5.runMayThrow(StorageProxy.java:552)
        at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1658)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)


nodetool status being executed on 3 old nodes, shows old ghost node:

Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns   Host ID                              
Rack
UN  10.14.128.109  2.8 GB     141     4.1%   32260392-12c2-4f1a-812e-87fd9a960d10 
RAC2
UN  10.24.33.187   2.12 GB    258     42.7%  ab66dd02-96b2-4504-8403-7d066f911698 
RAC3
UN  10.20.149.165  2.99 GB    251     4.5%   a0792f59-20b1-4017-a7f6-88e0c0d7f86f 
RAC1
DN  10.11.73.104   1.07 GB    2       1.0%   null                                 
RAC1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns   Host ID                              
Rack
UN  10.34.78.23     2.21 GB    117     0.9%   2acd3766-404d-4cdc-b3e3-7b3b95856f0e 
RAC1
UN  10.20.23.171   2.22 GB    255     46.8%  67421e3a-1dfc-48a0-88b3-c6dbd64dc9d8 
RAC1


Is it possible to fix those IP collisions ?


Thanks.
Mime
View raw message