cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Giampiero Recco (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-5571) Reject bootstrapping endpoints that are already in the ring with different gossip data
Date Fri, 11 Jul 2014 19:19:06 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14059198#comment-14059198
] 

Giampiero Recco edited comment on CASSANDRA-5571 at 7/11/14 7:18 PM:
---------------------------------------------------------------------

Thank you Brandon, you are right, Priam has been probably abusing this non-feature for a while
essentially starting the cluster in a non-conventional (less-coordinated) way.

My understanding is that this issue's patch breaks that, although not so much for the new
check that has been introduced but rather for the timing and the collaterals of this check
(i.e. other nodes ignoring the gossip message), and that's why I raised the problem here,
wondering if there is any way we can mitigate the collateral effects of this patch while Priam
improves the way it bootstrap a cluster.

Anyhow, I'm already trying to put this in the Priam dev agenda, and, as you suggested, I'll
try to discuss this also on the Cassandra ml.

Update: For reference here you can find one of the issues reporting the problem on the Priam
side: https://github.com/Netflix/Priam/issues/313



was (Author: grecco-gn):
Thank you Brandon, you are right, Priam has been probably abusing this non-feature for a while
essentially starting the cluster in a non-conventional (less-coordinated) way.

My understanding is that this issue's patch breaks that, although not so much for the new
check that has been introduced but rather for the timing and the collaterals of this check
(i.e. other nodes ignoring the gossip message), and that's why I raised the problem here,
wondering if there is any way we can mitigate the collateral effects of this patch while Priam
improves the way it bootstrap a cluster.

Anyhow, I'm already trying to put this in the Priam dev agenda, and, as you suggested, I'll
try to discuss this also on the Cassandra ml.


> Reject bootstrapping endpoints that are already in the ring with different gossip data
> --------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5571
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5571
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Rick Branson
>            Assignee: Tyler Hobbs
>             Fix For: 2.0.2
>
>         Attachments: 5571-2.0-v1.patch, 5571-2.0-v2.patch, 5571-2.0-v3.patch
>
>
> The ring can be silently broken by improperly bootstrapping an endpoint that has an existing
entry in the gossip table. In the case where a node attempts to bootstrap with the same IP
address as an existing ring member, the old token metadata is dropped without warning, resulting
in range shifts for the cluster.
> This isn't so bad for non-vnode cases where, in general, tokens are explicitly assigned,
and a bootstrap on the same token would result in no range shifts. For vnode cases, the convention
is to just let nodes come up by selecting their own tokens, and a bootstrap will override
the existing tokens for that endpoint.
> While there are some other issues open for adding an explicit rebootstrap feature for
vnode cases, given the changes in operator habits for vnode rings, it seems a bit too easy
to make this happen. Even more undesirable is the fact that it's basically silent.
> This is a proposal for checking for this exact case: bootstraps on endpoints with existing
ring entries that have different hostIDs and/or tokens should be rejected with an error message
describing what happened and how to override the safety check. It looks like the override
can be supported using the existing "nodetool removenode -force".
> I can work up a patch for this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message