cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fd Habash <fmhab...@gmail.com>
Subject RE: When Replacing a Node, How to Force a Consistent Bootstrap
Date Thu, 14 Dec 2017 14:33:23 GMT
“ … but it's better to repair before and after if possible …”

After, I simply run ‘nodetool repair –full’ on the replaced node. But before bootstrapping,
if my cluster is distributed over 3 AZ’s, what do I repair? The entire other AZ’s? As
one pointed out earlier, I can use ‘nodetool repair -hosts”, how do you identify what
specific hosts to repair?

Thanks 

----------------
Thank you

From: Fd Habash
Sent: Thursday, December 7, 2017 12:09 PM
To: user@cassandra.apache.org
Subject: RE: When Replacing a Node, How to Force a Consistent Bootstrap

Thank you.

How do I identify what other 2 nodes the former downed node replicated with? A replica set
of 3 nodes A,B,C. Now, C has been terminated by AWS and is gone. Using the getendpoints assumes
knowing a partition key value, but how do you even know what key to use?

If there is a way to identify A and B, I, then, can simply run ‘nodetool repair’ to repair
ALL the ranges on either.

Thanks 

----------------
Thank you

From: kurt greaves
Sent: Wednesday, December 6, 2017 6:45 PM
To: User
Subject: Re: When Replacing a Node, How to Force a Consistent Bootstrap

That's also an option but it's better to repair before and after if possible, if you don't
repair beforehand you could end up missing some replicas until you repair after replacement,
which could cause queries to return old/no data. Alternatively you could use ALL after replacing
until the repair completes.

For example, A and C have replica a, A dies, on replace A streams the partition owning a from
B, and thus is still inconsistent. QUORUM query hits A and B, and no results are returned
for a.

On 5 December 2017 at 23:04, Fred Habash <fmhabash@gmail.com> wrote:
Or, do a full repair after bootstrapping completes?



On Dec 5, 2017 4:43 PM, "Jeff Jirsa" <jjirsa@gmail.com> wrote:
You cant ask cassandra to stream from the node with the "most recent data", because for some
rows B may be most recent, and for others C may be most recent - you'd have to stream from
both (which we don't support).

You'll need to repair (and you can repair before you do the replace to avoid the window of
time where you violate consistency - use the -hosts option to allow repair with a down host,
you'll repair A+C, so when B starts it'll definitely have all of the data).


On Tue, Dec 5, 2017 at 1:38 PM, Fd Habash <fmhabash@gmail.com> wrote:
Assume I have cluster of 3 nodes (A,B,C). Row x was written with CL=LQ to node A and B. Before
it was written to C, node B crashes. I replaced B and it bootstrapped data from node C.
 
Now, row x is missing from C and B.  If node A crashes, it will be replaced and it will bootstrap
from either C or B. As such, row x is now completely gone from the entire ring. 
 
Is this scenario possible at all (at least in C* < 3.0). 
 
How can a newly replaced node be forced to bootstrap from the node in the replica set that
has the most recent data? 
 
Otherwise, we have to repair a node immediately after bootstrapping it for a node replacement.
 
Thank you
 





Mime
View raw message