Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Date: Fri, 2 Dec 2016 18:19:15 +0000 (UTC)
From: Anuj Wadehra <anujw_2003@yahoo.co.in>
Reply-To: "anujw_2003@yahoo.co.in" <anujw_2003@yahoo.co.in>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Message-ID: <2100223211.5426044.1480702755902@mail.yahoo.com>
Subject: Re: Single cluster node restore
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_5426043_995537969.1480702755900"
References: <2100223211.5426044.1480702755902.ref@mail.yahoo.com>
archived-at: Fri, 02 Dec 2016 18:19:34 -0000

------=_Part_5426043_995537969.1480702755900
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Petr,
If data corruption means accidental data deletions via Cassandra commands, =
you have to restore entire cluster with latest snapshots. This may lead to =
data loss as there may be valid updates after the snapshot was taken but be=
fore the data deletion. Restoring single node with snapshot wont help as Ca=
ssandra replicated the accidental deletes to all nodes.
If data corruption means accidental deletion of some sstable files from fil=
e system of a node, repair would fix it.
If data corruption means unreadable data due to hardware issues etc, you wi=
ll have two options after replacing the disk: bootstrap or restore snapshot=
 on the single affected node. If you have huge data per node e.g. 300Gb , y=
ou may want to restore from Snapshot followed by repair. Restoring snapshot=
 on single node is faster than streaming all data via bootstrap.=C2=A0If th=
e node is not recoverable and must be replaced, you should be able to do au=
to-boostrap or restore from snapshot with auto-bootstrap set to false. I ha=
vent replaced a dead node with snapshot but there should not be any issues =
as token ranges dont change when you replace a node.


Thanks
Anuj=20
=20
  On Tue, 29 Nov, 2016 at 11:08 PM, Petr Malik<pmalik@tesora.com> wrote:  =
=20


Hi.

I have a question about Cassandra backup-restore strategies.


As far as I understand Cassandra has been designed to survive hardware fail=
ures by relying on data replication.


It seems like people still want backup/restore for case when somebody accid=
entally deletes data or the data gets otherwise corrupted.

In that case restoring all keyspace/table snapshots on all nodes should bri=
ng it back.


I am asking because I often read directions on restoring a single node in a=
 cluster. I am just wondering under what circumstances could this be done s=
afely.


Please correct me if i am wrong but restoring just a single node does not r=
eally roll back the data as the newer (corrupt) data will be served by othe=
r replicas and eventually propagated to the restored node. Right?

In fact by doing so one may end up reintroducing deleted data back...


Also since Cassandra distributes the data throughout the cluster it is not =
clear on which mode any particular (corrupt) data resides and hence which t=
o restore.


I guess this is a long way of asking whether there is an advantage of tryin=
g to restore just a single node in a Cassandra cluster as opposed to say re=
placing the dead node and letting Cassandra handle the replication.


Thanks.
 =20

------=_Part_5426043_995537969.1480702755900
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Petr,<div id=3D"yMail_cursorElementTracker_1480701533977"><br></div><div=
 id=3D"yMail_cursorElementTracker_1480701533977">If data corruption means a=
ccidental data deletions via Cassandra commands, you have to restore entire=
 cluster with latest snapshots. This may lead to data loss as there may be =
valid updates after the snapshot was taken but before the data deletion. Re=
storing single node with snapshot wont help as Cassandra replicated the acc=
idental deletes to all nodes.<div id=3D"yMail_cursorElementTracker_14806114=
31134"><br>If data corruption means accidental deletion of some sstable fil=
es from file system of a node, repair would fix it.</div><div id=3D"yMail_c=
ursorElementTracker_1480611499451"><br></div><div id=3D"yMail_cursorElement=
Tracker_1480611501641">If data corruption means unreadable data due to hard=
ware issues etc, you will have two options after replacing the disk: bootst=
rap or restore snapshot on the single affected node. If you have huge data =
per node e.g. 300Gb , you may want to restore from Snapshot followed by rep=
air. Restoring snapshot on single node is faster than streaming all data vi=
a bootstrap.&nbsp;</div><div id=3D"yMail_cursorElementTracker_1480701703291=
">If the node is not recoverable and must be replaced, you should be able t=
o do auto-boostrap or restore from snapshot with auto-bootstrap set to fals=
e. I havent replaced a dead node with snapshot but there should not be any =
issues as token ranges dont change when you replace a node.<br></div><div i=
d=3D"yMail_cursorElementTracker_1480701517692"><br></div><div id=3D"yMail_c=
ursorElementTracker_1480701517861"><br></div><div id=3D"yMail_cursorElement=
Tracker_1480612022905"><br><div id=3D"ymail_android_signature">Thanks<br>An=
uj</div> <br> <blockquote style=3D"margin: 0 0 20px 0;"> <header style=3D"f=
ont-family:Roboto, sans-serif; color:#6D00F6;"> <div>On Tue, 29 Nov, 2016 a=
t 11:08 PM, Petr Malik</div><div>&lt;pmalik@tesora.com&gt; wrote:</div> </h=
eader> <div style=3D"padding: 10px 0 0 20px; margin: 10px 0 0 0; border-lef=
t: 1px solid #6D00F6;">=20
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri, Arial, Helvetica, sans-serif;" dir=3D"ltr">
<p><br>
</p>
<div id=3D"divtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt;color:=
#000000;font-family:Calibri, Arial, Helvetica, sans-serif;">
<div>Hi.<br>
<br>
I have a question about Cassandra backup-restore strategies.<br>
<br>
<p dir=3D"ltr">As far as I understand Cassandra has been designed to surviv=
e hardware failures by relying on data replication.</p>
<p dir=3D"ltr"><br>
</p>
<p dir=3D"ltr">It seems like people still want backup/restore for case when=
 somebody accidentally deletes data or the data gets otherwise corrupted.</=
p>
<p dir=3D"ltr">In that case restoring all keyspace/table snapshots on all n=
odes should bring it back.</p>
<p dir=3D"ltr"><br>
</p>
<p dir=3D"ltr">I am asking because I often read directions on restoring a s=
ingle node in a cluster. I am just wondering under what circumstances could=
 this be done safely.
<br>
</p>
<p dir=3D"ltr"><br>
</p>
<p dir=3D"ltr">Please correct me if i am wrong but restoring just a single =
node does not really roll back the data as the newer (corrupt) data will be=
 served by other replicas and eventually propagated to the restored node. R=
ight?</p>
<p dir=3D"ltr">In fact by doing so one may end up reintroducing deleted dat=
a back...</p>
<p dir=3D"ltr"><br>
</p>
<p dir=3D"ltr">Also since Cassandra distributes the data throughout the clu=
ster it is not clear on which mode any particular (corrupt) data resides an=
d hence which to restore.</p>
<p dir=3D"ltr"><br>
</p>
<p dir=3D"ltr">I guess this is a long way of asking whether there is an adv=
antage of trying to restore just a single node in a Cassandra cluster as op=
posed to say replacing the dead node and letting Cassandra handle the repli=
cation.</p>
<p dir=3D"ltr"><br>
</p>
<p dir=3D"ltr">Thanks.</p>
</div>
</div>
</div>
 </div> </blockquote></div></div>
------=_Part_5426043_995537969.1480702755900--