cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Coli (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-5571) Reject bootstrapping endpoints that are already in the ring with different gossip data
Date Thu, 30 Jan 2014 01:05:07 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886113#comment-13886113
] 

Robert Coli edited comment on CASSANDRA-5571 at 1/30/14 1:04 AM:
-----------------------------------------------------------------

To anyone using vnodes and looking to reduce their exposure to this before 2.0.2, you have
two obvious workarounds :

1) manually pick tokens, and bootstrap your new vnode node with all 256 of them in in intiial_token
comma delimited list

OR

1) continue to let num_tokens pick tokens for you and boostrap
2) collect all this node's tokens into a comma delimited list with a command like
{noformat}
IP=__NODE_IP__ ; nodetool ring | fgrep -w "$IP" | awk '{print $NF}' |xargs -d,

or

IP =__NODE_IP__ ; nodetool -h $IP info -T |grep Token | awk '{print $NF}' | tr '\n' ','
{noformat}
3) put the comma delimited list from 2 into the initial_token line in cassandra.yaml before
the next time the node restarts


was (Author: rcoli):
To anyone using vnodes and looking to reduce their exposure to this before 2.0.2, you have
two obvious workarounds :

1) manually pick tokens, and bootstrap your new vnode node with all 256 of them in in intiial_token
comma delimited list

OR

1) continue to let num_tokens pick tokens for you and boostrap
2) collect all this node's tokens into a comma delimited list with a command like
{noformat}
IP=__NODE_IP__ ; nodetool ring | fgrep -w "$IP" | awk '{print $NF}' |xargs -d,
{noformat}
3) put the comma delimited list from 2 into the initial_token line in cassandra.yaml before
the next time the node restarts

> Reject bootstrapping endpoints that are already in the ring with different gossip data
> --------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5571
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5571
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Rick Branson
>            Assignee: Tyler Hobbs
>             Fix For: 2.0.2
>
>         Attachments: 5571-2.0-v1.patch, 5571-2.0-v2.patch, 5571-2.0-v3.patch
>
>
> The ring can be silently broken by improperly bootstrapping an endpoint that has an existing
entry in the gossip table. In the case where a node attempts to bootstrap with the same IP
address as an existing ring member, the old token metadata is dropped without warning, resulting
in range shifts for the cluster.
> This isn't so bad for non-vnode cases where, in general, tokens are explicitly assigned,
and a bootstrap on the same token would result in no range shifts. For vnode cases, the convention
is to just let nodes come up by selecting their own tokens, and a bootstrap will override
the existing tokens for that endpoint.
> While there are some other issues open for adding an explicit rebootstrap feature for
vnode cases, given the changes in operator habits for vnode rings, it seems a bit too easy
to make this happen. Even more undesirable is the fact that it's basically silent.
> This is a proposal for checking for this exact case: bootstraps on endpoints with existing
ring entries that have different hostIDs and/or tokens should be rejected with an error message
describing what happened and how to override the safety check. It looks like the override
can be supported using the existing "nodetool removenode -force".
> I can work up a patch for this.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message