cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tyler Hobbs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5916) gossip and tokenMetadata get hostId out of sync on failed replace_node with the same IP address
Date Tue, 08 Oct 2013 21:43:43 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789734#comment-13789734
] 

Tyler Hobbs commented on CASSANDRA-5916:
----------------------------------------

I'm testing this out with a three-node ccm cluster.  If I do the following:
# (optional) stop node3
# add a blank node4
# start node4 with replace_address=127.0.0.3

I'll get the following:
{noformat}
ERROR 16:29:02,689 Exception encountered during startup
java.lang.RuntimeException: Cannot replace_address /127.0.0.3because it doesn't exist in gossip
    at org.apache.cassandra.service.StorageService.prepareReplacementInfo(StorageService.java:421)
    at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:623)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:604)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:501)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
java.lang.RuntimeException: Cannot replace_address /127.0.0.3because it doesn't exist in gossip
    at org.apache.cassandra.service.StorageService.prepareReplacementInfo(StorageService.java:421)
    at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:623)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:604)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:501)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
Exception encountered during startup: Cannot replace_address /127.0.0.3because it doesn't
exist in gossip
ERROR 16:29:02,692 Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
    at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321)
    at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370)
    at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88)
    at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:569)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at java.lang.Thread.run(Thread.java:724)
{noformat}

This happens whether node3 is up or down.  It seems like this problem occurs any time replace_address
doesn't match the broadcast address.

> gossip and tokenMetadata get hostId out of sync on failed replace_node with the same
IP address
> -----------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5916
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5916
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Brandon Williams
>            Assignee: Brandon Williams
>             Fix For: 1.2.11
>
>         Attachments: 5916.txt, 5916-v2.txt
>
>
> If you try to replace_node an existing, live hostId, it will error out.  However if you're
using an existing IP to do this (as in, you chose the wrong uuid to replace on accident) then
the newly generated hostId wipes out the old one in TMD, and when you do try to replace it
replace_node will complain it does not exist.  Examination of gossipinfo still shows the old
hostId, however now you can't replace it either.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message