cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Furlong <mfurl...@ancestry.com>
Subject RE: Node failure
Date Fri, 06 Oct 2017 18:26:49 GMT
I’ll check to see what our app is using.

Thanks
Mark
801-705-7115 office

From: Steinmaurer, Thomas [mailto:thomas.steinmaurer@dynatrace.com]
Sent: Friday, October 6, 2017 12:25 PM
To: user@cassandra.apache.org
Subject: RE: Node failure

QUORUM should succeed with a RF=3 and 2 of 3 nodes available.

Modern client drivers also have ways to “downgrade” the CL of requests, in case they fail.
E.g. for the Java driver: http://docs.datastax.com/en/latest-java-driver-api/com/datastax/driver/core/policies/DowngradingConsistencyRetryPolicy.html


Thomas

From: Mark Furlong [mailto:mfurlong@ancestry.com]
Sent: Freitag, 06. Oktober 2017 19:43
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: Node failure

Thanks for the detail. I’ll have to remove and then add one back in. It’s my consistency
levels that may bite me in the interim.

Thanks
Mark
801-705-7115 office

From: Jeff Jirsa [mailto:jjirsa@gmail.com]
Sent: Friday, October 6, 2017 11:29 AM
To: cassandra <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Node failure

There's a lot to talk about here, what's your exact question?


- You can either remove it from the cluster or replace it. You typically remove it if it'll
never be replaced, but in RF=3 with 3 nodes, you probably need to replace it. To replace,
you'll start a new server with -Dcassandra.replace_address=a.b.c.d ( http://cassandra.apache.org/doc/latest/operating/topo_changes.html#replacing-a-dead-node
) , and it'll stream data from the neighbors and eventually replace the dead node in the ring
(the dead node will be removed from 'nodetool status', the new node will be there instead).

- If you're not going to replace it, things get a bit more complex - you'll do some combination
of repair, 'nodetool removenode' or 'nodetool assassinate', and ALTERing the keyspace to set
RF=2. The order matters, and so does the consistency level you use for reads/writes (so we
can tell you whether or not you're likely to lose data in this process), so I'm not giving
step-by-steps here because it's not very straight forward and there are a lot of caveats.




On Fri, Oct 6, 2017 at 10:20 AM, Mark Furlong <mfurlong@ancestry.com<mailto:mfurlong@ancestry.com>>
wrote:
What happens when I have a 3 node cluster with RF 3 and a node fails that needs to be removed?

Mark Furlong

Sr. Database Administrator

mfurlong@ancestry.com<mailto:mfurlong@ancestry.com>
M: 801-859-7427<tel:(801)%20859-7427>
O: 801-705-7115<tel:(801)%20705-7115>
1300 W Traverse Pkwy<https://maps.google.com/?q=1300+W+Traverse+Pkwy%0D+Lehi,+UT+84043&entry=gmail&source=g>
Lehi, UT 84043<https://maps.google.com/?q=1300+W+Traverse+Pkwy%0D+Lehi,+UT+84043&entry=gmail&source=g>





​[http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]




The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
Mime
View raw message