incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Coli <rc...@eventbrite.com>
Subject Re: Safely adding new nodes without losing data
Date Tue, 23 Jul 2013 16:48:40 GMT
  On Sat, Jul 20, 2013 at 7:30 AM, E S <tr1sklion@yahoo.com> wrote:

> I am trying to understand the best procedure for adding new nodes.  The
> one that I see most often online seems to have a hole where there is a low
> probability of permanently losing data.  I want to understand what I am
> missing in my understanding.
>

You aren't missing anything. Congratulations, you have deduced the
following bug :

https://issues.apache.org/jira/browse/CASSANDRA-2434

"Range movements violate consistency"

 I don't think this problem would normally happen without vnodes, because
> when doubling you would alternate the new nodes with the old nodes in the
> ring, so while quorum might not work until the final repair, "all" would,
> and a repair would solve the problem.  With vnodes though, some of the
> ranges will follow the pattern above (range ownership moving from A,B,C to
> D,E,F).
>

I believe this is correct, though adding nodes "between" other nodes is
just convention, there is nothing actually keeping you from being exposed
to 2434 if you do not do this.


> Am I missing something here?  If I'm right, I think the only way to avoid
> this is adding less then a quorum of new nodes (in this case 1) before
> doing a repair.  That would be painful since repairs take a while.
>

If you read/write at QUORUM, this is true. If you read/write at ONE, my
reading of 2434 is that any range movement is capable of losing data
because your data may be only on the node which has lost the range at the
end of the movement, which is not necessarily the host streaming its keys
to the bootstrapping node. Even running a repair before the movement only
reduces the chance of loss, it does not eliminate it.

=Rob

Mime
View raw message