cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean Tremblay <jean.tremb...@zen-innovations.com>
Subject Re: Restore Snapshots
Date Fri, 26 Jun 2015 06:44:11 GMT
Good morning,
Alain, thank you so much. This is exactly what I needed.

 In my test I had a node which had for whatever reason the directory containing my data corrupted.
I keep in a separate folder my snapshots.

Here are the steps I took to recover my sick node:

0) Cassandra is stopped on my sick node.
1) I wiped out my data directory. My snapshots were kept outside this directory.
2) I modified my Cassandra.yaml. I added auto_bootstrap: false .This is to make sure that
my node does not synch with the others.
3) I restarted Cassandra. This step created a basic structure for my new data directory.
4) I did the command: nodetool resetlocalschema. This recreated all the folders for my cf.
5) I stopped Cassandra on my node.
6) I copied my snapshot in the right location. I actually hard linked them, this is very fast.
7) I restarted Cassandra.

That's it.

Thank you SO MUCH ALAIN for your support. You really helped me a lot.
On 25 Jun,2015, at 18:37, Alain RODRIGUEZ <arodrime@gmail.com<mailto:arodrime@gmail.com>>
wrote:

Hi Jean,

Answers in line to be sure to be exhaustive:

- how can I restore the data directory structure in order to copy my snapshots at the right
position?
--> making a script to do it and testing it I would say. basically under any table repo
you have a "snapshots/snapshot_name" directory (snapshot_name is timestamp if not specified
off the top of my head..) and then your sstables.

- is it possible to recreate the schema on one node?
--> The easiest way that come to my mind is to set "auto_bootstrap: false" on a node not
already in the ring. If you have trouble with the schema of a node in the ring run a "nodetool
resetlocalschema"

- how can I avoid the node from streaming from the other nodes?
--> See above (auto_bootstrap: false). BTW, option might not be present at all, just add
it.

- must I also have the snapshot of the system tables in order to restore a node from only
the snapshot of my tables?
--> just you user table. Yet remember that snapshot is per node and as such you will just
have part of the data this node use to hold. meaning that if the new node have different tokens,
there will be unused data + missing data for sure.

Basically when a node is down I use to remove it, repair the cluster, and bootstap it (auto_bootstrap:
true). Streams are part of Cassandra. I accept that. An other solution would be to "replace"
the node --> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html

C*heers,

Alain

2015-06-25 17:07 GMT+02:00 Jean Tremblay <jean.tremblay@zen-innovations.com<mailto:jean.tremblay@zen-innovations.com>>:
Hi,

I am testing snapshot restore procedures in case of a major catastrophe on our cluster. I'm
using Cassandra 2.1.7 with RF:3

The scenario that I am trying to solve is how to quickly get one node back to work after its
disk failed and lost all its data assuming that the only thing I have is its snapshots.

The procedure that I'm following is the one explained here: http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html

I can do a snapshot that is straight forward.
My problem is in the restore of the snapshot.

If I restart Cassandra with an empty data directory the node will bootstrap.
Bootstrap is very nice, since it recreate the schema and reload the data from its neighbour.
But this is quite heavy traffic and quite a slow process.

My questions are:

- how can I restore the data directory structure in order to copy my snapshots at the right
position?
- is it possible to recreate the schema on one node?
- how can I avoid the node from streaming from the other nodes?
- must I also have the snapshot of the system tables in order to restore a node from only
the snapshot of my tables?

Thanks for your comments.

Jean






Mime
View raw message