cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anuj Wadehra <>
Subject Re: Single cluster node restore
Date Fri, 02 Dec 2016 18:19:15 GMT
Hi Petr,
If data corruption means accidental data deletions via Cassandra commands, you have to restore
entire cluster with latest snapshots. This may lead to data loss as there may be valid updates
after the snapshot was taken but before the data deletion. Restoring single node with snapshot
wont help as Cassandra replicated the accidental deletes to all nodes.
If data corruption means accidental deletion of some sstable files from file system of a node,
repair would fix it.
If data corruption means unreadable data due to hardware issues etc, you will have two options
after replacing the disk: bootstrap or restore snapshot on the single affected node. If you
have huge data per node e.g. 300Gb , you may want to restore from Snapshot followed by repair.
Restoring snapshot on single node is faster than streaming all data via bootstrap. If the
node is not recoverable and must be replaced, you should be able to do auto-boostrap or restore
from snapshot with auto-bootstrap set to false. I havent replaced a dead node with snapshot
but there should not be any issues as token ranges dont change when you replace a node.

  On Tue, 29 Nov, 2016 at 11:08 PM, Petr Malik<> wrote:   


I have a question about Cassandra backup-restore strategies.

As far as I understand Cassandra has been designed to survive hardware failures by relying
on data replication.

It seems like people still want backup/restore for case when somebody accidentally deletes
data or the data gets otherwise corrupted.

In that case restoring all keyspace/table snapshots on all nodes should bring it back.

I am asking because I often read directions on restoring a single node in a cluster. I am
just wondering under what circumstances could this be done safely.

Please correct me if i am wrong but restoring just a single node does not really roll back
the data as the newer (corrupt) data will be served by other replicas and eventually propagated
to the restored node. Right?

In fact by doing so one may end up reintroducing deleted data back...

Also since Cassandra distributes the data throughout the cluster it is not clear on which
mode any particular (corrupt) data resides and hence which to restore.

I guess this is a long way of asking whether there is an advantage of trying to restore just
a single node in a Cassandra cluster as opposed to say replacing the dead node and letting
Cassandra handle the replication.


View raw message