cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Durity, Sean R" <SEAN_R_DUR...@homedepot.com>
Subject RE: Recover lost node from backup or evict/re-add?
Date Wed, 12 Jun 2019 18:17:04 GMT
I’m not sure it is correct to say, “you cannot.” However, that is a more complicated
restore and more likely to lead to inconsistent data and take longer to do. You are basically
trying to start from a backup point and roll everything forward and catch up to current.

Replacing/re-streaming is the well-trodden path. You are getting the net result of all that
has happened since the node failure. And the node is not returning data to the clients while
the bootstrap is running. If you have a restored/repairing node, it will accept client (and
coordinator) connections even though it isn’t (guaranteed) consistent, yet.

As I understand it – a full cluster recovery from backup still requires repair across the
cluster to ensure consistency. In my experience, most apps cannot wait for a full restore/repair.
Availability matters more. They also don’t want to pay for even more disk to hold some level
of backups.

There are some companies that provide finer-grained backup and recovery options, though.

Sean Durity

From: Alan Gano <AGano@tsys.com>
Sent: Wednesday, June 12, 2019 1:43 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] RE: Recover lost node from backup or evict/re-add?


Is it correct to say that a lost node cannot be restored from backup?  You must either replace
the node or evict/re-add (i.e., rebuild from other nodes).

Also, that snapshot, incremental, commitlog backups are relegated to application keyspace
recovery only?


How about recovery of the entire cluster? (rolling it back).  Are snapshots exact enough,
in time, to not have a nodes that differ, in point-in-time, from the rest of the cluster?
 Would those nodes be recoverable (nodetool repair?) … which brings me back to recovering
a lost node from backup (restore last snapshot, and run nodetool repair?).


Thanks,

Alan Gano


From: Jeff Jirsa [mailto:jjirsa@gmail.com]
Sent: Wednesday, June 12, 2019 10:14 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Recover lost node from backup or evict/re-add?

A host can replace itself using the method I described

On Jun 12, 2019, at 7:10 AM, Alan Gano <AGano@tsys.com<mailto:AGano@tsys.com>>
wrote:
I guess I’m considering this scenario:

  *   host and configuration have survived
  *   /data is gone
  *   /backups have survived

I have tested recovering from this scenario with an evict/re-add, which worked fine.

If I restore from backup, the node will be behind the cluster – errrr, does it get caught
up after a restore and start it up?

Alan

From: Jeff Jirsa [mailto:jjirsa@gmail.com]
Sent: Wednesday, June 12, 2019 10:02 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Recover lost node from backup or evict/re-add?

To avoid violating consistency guarantees, you have to repair the replicas while the lost
node is down

Once you do that it’s typically easiest to bootstrap a replacement (there’s a property
named “replace address first boot” you can google or someone can link) that tells a new
joining host to take over for a failed machine.


On Jun 12, 2019, at 6:54 AM, Alan Gano <AGano@tsys.com<mailto:AGano@tsys.com>>
wrote:

If I lose a node, does it make sense to even restore from snapshot/incrementals/commitlogs?

Or is the best way to do an evict/re-add?


Thanks,

Alan.

NOTICE: This communication is intended only for the person or entity to whom it is addressed
and may contain confidential, proprietary, and/or privileged material. Unless you are the
intended addressee, any review, reliance, dissemination, distribution, copying or use whatsoever
of this communication is strictly prohibited. If you received this in error, please reply
immediately and delete the material from all computers. Email sent through the Internet is
not secure. Do not use email to send us confidential information such as credit card numbers,
PIN numbers, passwords, Social Security Numbers, Account numbers, or other important and confidential
information.
NOTICE: This communication is intended only for the person or entity to whom it is addressed
and may contain confidential, proprietary, and/or privileged material. Unless you are the
intended addressee, any review, reliance, dissemination, distribution, copying or use whatsoever
of this communication is strictly prohibited. If you received this in error, please reply
immediately and delete the material from all computers. Email sent through the Internet is
not secure. Do not use email to send us confidential information such as credit card numbers,
PIN numbers, passwords, Social Security Numbers, Account numbers, or other important and confidential
information.
NOTICE: This communication is intended only for the person or entity to whom it is addressed
and may contain confidential, proprietary, and/or privileged material. Unless you are the
intended addressee, any review, reliance, dissemination, distribution, copying or use whatsoever
of this communication is strictly prohibited. If you received this in error, please reply
immediately and delete the material from all computers. Email sent through the Internet is
not secure. Do not use email to send us confidential information such as credit card numbers,
PIN numbers, passwords, Social Security Numbers, Account numbers, or other important and confidential
information.

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is
intended solely for the addressee. Access to this Email by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject to the terms and
conditions expressed in any applicable governing The Home Depot terms of business or client
engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy
and content of this attachment and for any damages or losses arising from any inaccuracies,
errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature,
which may be contained in this attachment and shall not be liable for direct, indirect, consequential
or special damages in connection with this e-mail message or its attachment.
Mime
View raw message