cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tyler Hobbs (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-2434) range movements can violate consistency
Date Sat, 22 Feb 2014 00:42:19 GMT


Tyler Hobbs commented on CASSANDRA-2434:

Okay, with the workaround on the FD, bootstrap seems to work.  Do we want to split that fix
into a separate ticket?

However, relocate seems to be seriously broken.  With a three node cluster and one of the
nodes down, I can make relocate fail in a couple of ways:
* {{oldEndpoints}} == {{newEndpoints}}, so the assertion that the difference between them
has length 1 fails
* There are no ranges that contain the "desiredRange", resulting in the IllegalStateException
being thrown ("No sources found for " + toFetch);

With that said, nothing (including the tools) uses relocate.  The JMX version doesn't work
with jconsole, so I had to add a method to test this.  I'm not even sure that relocate worked
before this patch for vnodes, because there's only minimal test coverage for relocate.  IMO,
we shouldn't even try to modify this without good test coverage.  But if nothing even uses
relocate... I'm not sure what to do.  Thoughts?

> range movements can violate consistency
> ---------------------------------------
>                 Key: CASSANDRA-2434
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Peter Schuller
>            Assignee: T Jake Luciani
>             Fix For: 2.1 beta2
>         Attachments: 2434-3.patch.txt, 2434-testery.patch.txt
> My reading (a while ago) of the code indicates that there is no logic involved during
bootstrapping that avoids consistency level violations. If I recall correctly it just grabs
neighbors that are currently up.
> There are at least two issues I have with this behavior:
> * If I have a cluster where I have applications relying on QUORUM with RF=3, and bootstrapping
complete based on only one node, I have just violated the supposedly guaranteed consistency
semantics of the cluster.
> * Nodes can flap up and down at any time, so even if a human takes care to look at which
nodes are up and things about it carefully before bootstrapping, there's no guarantee.
> A complication is that not only does it depend on use-case where this is an issue (if
all you ever do you do at CL.ONE, it's fine); even in a cluster which is otherwise used for
QUORUM operations you may wish to accept less-than-quorum nodes during bootstrap in various
emergency situations.
> A potential easy fix is to have bootstrap take an argument which is the number of hosts
to bootstrap from, or to assume QUORUM if none is given.
> (A related concern is bootstrapping across data centers. You may *want* to bootstrap
to a local node and then do a repair to avoid sending loads of data across DC:s while still
achieving consistency. Or even if you don't care about the consistency issues, I don't think
there is currently a way to bootstrap from local nodes only.)
> Thoughts?

This message was sent by Atlassian JIRA

View raw message