cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-5916) gossip and tokenMetadata get hostId out of sync on failed replace_node with the same IP address
Date Wed, 16 Oct 2013 20:26:46 GMT


Brandon Williams updated CASSANDRA-5916:

    Attachment: 5916-v3.txt

There are two distinct cases here: we replace 'ourself' with the same IP, or we replace a
dead node with a new IP (ala ec2.)  We can't know which one we're doing _a priori_, so we
shadow gossip.  If we're replacing the same IP, our shadow SYN will contain it, and the remote
node will ACK with what we need.

If we're _not_ replacing with the same IP, there's a problem: an ACK will only contain what
was present in the SYN digest list.  One could argue this is the sender being naive, since
it obviously knows the node that sent the SYN doesn't have some states that it does, but I
think at scale this makes sense since it's possible a third node has begun gossiping with
the SYN sender, too.  In any case, I don't want to change that behavior at this point.

The other problem is, we can't just sit around and wait for someone to send us a populated
SYN either, since we're not a part of gossip and we're new.  But we don't know we're new yet,
and can't insert ourselves into gossip either, or we'll break the case of using the same IP.

So, we'll create a special case for shadow gossip, and redefine it a bit.  Instead of sending
a SYN with our own endpoint and a generation of zero, we'll send a completely empty SYN (digest-wise,
we still populate the cluster name and partioner, since those checks still make sense.)  This
won't ever normally occur in gossip, because a node always knows about and adds itself.  When
we see an empty SYN, we can know that the node that sent it is asking for everything we've
got, and we can ACK with just that, allowing the replacement node to have whatever it needs
for either the same or different IP cases.

v3 does this.

> gossip and tokenMetadata get hostId out of sync on failed replace_node with the same
IP address
> -----------------------------------------------------------------------------------------------
>                 Key: CASSANDRA-5916
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Brandon Williams
>            Assignee: Brandon Williams
>             Fix For: 1.2.12
>         Attachments: 5916.txt, 5916-v2.txt, 5916-v3.txt
> If you try to replace_node an existing, live hostId, it will error out.  However if you're
using an existing IP to do this (as in, you chose the wrong uuid to replace on accident) then
the newly generated hostId wipes out the old one in TMD, and when you do try to replace it
replace_node will complain it does not exist.  Examination of gossipinfo still shows the old
hostId, however now you can't replace it either.

This message was sent by Atlassian JIRA

View raw message