cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juho Mäkinen (JIRA) <>
Subject [jira] [Commented] (CASSANDRA-7292) Can't seed new node into ring with (public) ip of an old node
Date Fri, 23 May 2014 16:49:02 GMT


Juho Mäkinen commented on CASSANDRA-7292:

I gave a quick read on CASSANDRA-6523 and I'm still digesting the ideas and info on that.
To begin with, my seed list contains two public/broadcast ips from the cluster. I haven't
tried specifying private addresses to the seed list, but I can do that on Monday if you feel
if that might help. Anyway I agree that the error messages could be made much more informative.

> Can't seed new node into ring with (public) ip of an old node
> -------------------------------------------------------------
>                 Key: CASSANDRA-7292
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 2.0.7, Ec2MultiRegionSnitch
>            Reporter: Juho Mäkinen
>              Labels: bootstrap, gossip
>         Attachments: cassandra-replace-address.log
> This bug prevents node to return with bootstrap into the cluster with its old ip.
> Scenario: five node ec2 cluster spread into three AZ, all in one region. I'm using Ec2MultiRegionSnitch.
Nodes are reported with their public ips (as Ec2MultiRegionSnitch requires)
> I simulated a loss of one node by terminating one instance. nodetool status reported
correctly that node was down. Then I launched new instance with the old public ip (i'm using
elastic ips) with "Dcassandra.replace_address=IP_ADDRESS" but the new node can't join the
>  INFO 07:20:43,424 Gathering node replacement information for /
>  INFO 07:20:43,428 Starting Messaging Service on port 9043
>  INFO 07:20:43,489 Handshaking version with /
>  INFO 07:20:43,491 Handshaking version with /
> (some delay)
> ERROR 07:21:14,445 Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds
> 	at org.apache.cassandra.gms.Gossiper.doShadowRound(
> 	at org.apache.cassandra.service.StorageService.prepareReplacementInfo(
> 	at org.apache.cassandra.service.StorageService.prepareToJoin(
> 	at org.apache.cassandra.service.StorageService.initServer(
> 	at org.apache.cassandra.service.StorageService.initServer(
> 	at org.apache.cassandra.service.CassandraDaemon.setup(
> 	at org.apache.cassandra.service.CassandraDaemon.activate(
> 	at org.apache.cassandra.service.CassandraDaemon.main(
> It does not help if I remove the "Dcassandra.replace_address=IP_ADDRESS" system property.

> Also it does not help to remove the node with "nodetool removenode" with or without the
cassandra.replace_address property.
> I think this is because the node information is preserved in the gossip info as seen
this output of "nodetool gossipinfo"
> /
>   DC:us-east
>   REMOVAL_COORDINATOR:REMOVER,d581309a-8610-40d4-ba30-cb250eda22a8
>   STATUS:removed,19311925-46b5-4fe4-928a-321e8adb731d,1401089960664
>   HOST_ID:19311925-46b5-4fe4-928a-321e8adb731d
>   SCHEMA:226f9315-b4b2-32c1-bfe1-f4bb49fccfd5
>   RACK:1b
>   LOAD:7.075290515E9
>   SEVERITY:0.0

This message was sent by Atlassian JIRA

View raw message