Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <CAJXyu8mJC-b3KL0_ZgcM7jVoPiNW6vKeUU+1Wq=zp8mBdTUTWQ@mail.gmail.com>
References: <CAJXyu8mJC-b3KL0_ZgcM7jVoPiNW6vKeUU+1Wq=zp8mBdTUTWQ@mail.gmail.com>
From: Alain RODRIGUEZ <arodrime@gmail.com>
Date: Thu, 12 Jan 2017 14:05:03 +0100
Message-ID: <CA+VSrLoOOLFCJTPnKmzh8SfqUkyZYZMar=xj0iR7XvZoyUpTyQ@mail.gmail.com>
Subject: Re: Check snapshot / sstable integrity
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=f403043612f0cbc3290545e55ed7
archived-at: Thu, 12 Jan 2017 13:05:34 -0000

--f403043612f0cbc3290545e55ed7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi J=C3=A9r=C3=B4me,

About this concern:

But my Op retains my arm and asks: "Are you sure that the snapshot is safe
> and will be restored before truncating data we have?"


Make sure to enable snapshot on truncate (cassandra.yaml) or do it
manually. This way if the restored dataset is worst than the current one
(the one you plan to truncate), you can always rollback this truncate /
restore action. This way you can tell your "Op" that this is perfectly safe
anyway, no data would be lost, even in the worst case scenario (not
considering the downtime that would be induced). Plus this snapshot is
cheap (hard links) and do not need to be moved around or kept once you are
sure the old backup fits your need.

Truncate is definitely the way to go before restoring a backup. Parsing the
data to delete it all is not really an option imho.

Then about the technical question "how to know that a snapshot is clean" it
would be good to define "clean". You can make sure the backup is readable,
consistent enough and correspond to what you want by inserting all  the
sstables into a testing cluster and performing some reads there before
doing it in production. You can use for example AWS EC2 machines with big
EBS attached or whatever and use the sstableloader to load data into it.

If you are just worried about SSTables format validity, there is no tool I
am aware of to check sstables well formatted but it might exist or be
doable. An other option might be to do a checksum on each sstable before
uploading it elsewhere and make sure it matches when downloaded back.
That's the first things that come to my mind.

Hope that is helpful. Hopefully, someone else will be able to point you to
an existing tool to do this work.

Cheers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2017-01-12 11:33 GMT+01:00 J=C3=A9r=C3=B4me Mainaud <jerome@mainaud.com>:

> Hello,
>
> Is there any tool to test the integrity of a snapshot?
>
> Suppose I have a snapshot based backup stored in an external low cost
> storage system that I want to restore to a database after someone deleted
> important data by mistake.
>
> Before restoring the files, I will truncate the table to remove the
> problematic tombstones.
>
> But my Op retains my arm and asks: "Are you sure that the snapshot is saf=
e
> and will be restored before truncating data we have?"
>
> If this scenario is a theoretical, the question is good. How can I verify
> that a snapshot is clean?
>
> Thank you,
>
> --
> J=C3=A9r=C3=B4me Mainaud
> jerome@mainaud.com
>

--f403043612f0cbc3290545e55ed7
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi J=C3=A9r=C3=B4me,<div><br></div><div>About this concern=
:</div><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px=
 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><sp=
an style=3D"font-size:12.8px">But my Op retains my arm and asks: &quot;Are =
you sure that the snapshot is safe and will be restored before truncating d=
ata we have?&quot;</span></blockquote><div><br></div><div>Make sure to enab=
le snapshot on truncate (cassandra.yaml) or do it manually. This way if the=
 restored dataset is worst than the current one (the one you plan to trunca=
te), you can always rollback this truncate / restore action. This way you c=
an tell your &quot;Op&quot; that this is perfectly safe anyway, no data wou=
ld be lost, even in the worst case scenario (not considering the downtime t=
hat would be induced). Plus this snapshot is cheap (hard links) and do not =
need to be moved around or kept once you are sure the old backup fits your =
need.</div><div><br></div><div>Truncate is definitely the way to go before =
restoring a backup. Parsing the data to delete it all is not really an opti=
on imho.<br><br>Then about the technical question &quot;how to know that a =
snapshot is clean&quot; it would be good to define &quot;clean&quot;. You c=
an make sure the backup is readable, consistent enough and correspond to wh=
at you want by inserting all =C2=A0the sstables into a testing cluster and =
performing some reads there before doing it in production. You can use for =
example AWS EC2 machines with big EBS attached or whatever and use the ssta=
bleloader to load data into it.=C2=A0</div><div><br></div><div>If you are j=
ust worried about SSTables format validity, there is no tool I am aware of =
to check sstables well formatted but it might exist or be doable. An other =
option might be to do a checksum on each sstable before uploading it elsewh=
ere and make sure it matches when downloaded back. That&#39;s the first thi=
ngs that come to my mind.</div><div><br></div><div>Hope that is helpful. Ho=
pefully, someone else will be able to point you to an existing tool to do t=
his work.</div><div><br></div><div>Cheers,</div><div><div>-----------------=
------</div><div>Alain Rodriguez - @arodream - <a href=3D"mailto:alain@thel=
astpickle.com">alain@thelastpickle.com</a></div><div>France</div><div><br><=
/div><div>The Last Pickle - Apache Cassandra Consulting</div><div><a href=
=3D"http://www.thelastpickle.com">http://www.thelastpickle.com</a></div></d=
iv></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">2017-01-=
12 11:33 GMT+01:00 J=C3=A9r=C3=B4me Mainaud <span dir=3D"ltr">&lt;<a href=
=3D"mailto:jerome@mainaud.com" target=3D"_blank">jerome@mainaud.com</a>&gt;=
</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo=
rder-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div>=
<div><div>Hello,<br><br></div>Is there any tool to test the integrity of a =
snapshot?<br><br></div>Suppose I have a snapshot based backup stored in an =
external low cost storage system that I want to restore to a database after=
 someone deleted important data by mistake.<br><br></div>Before restoring t=
he files, I will truncate the table to remove the problematic tombstones.<b=
r><br></div>But my Op retains my arm and asks: &quot;Are you sure that the =
snapshot is safe and will be restored before truncating data we have?&quot;=
<br><br></div>If this scenario is a theoretical, the question is good. How =
can I verify that a snapshot is clean?<br><br><div><div><div><div><div><div=
>Thank you,<span class=3D"HOEnZb"><font color=3D"#888888"><br></font></span=
></div><span class=3D"HOEnZb"><font color=3D"#888888"><div><div><br clear=
=3D"all"><div><div><div class=3D"m_-1193228424878089440gmail_signature" dat=
a-smartmail=3D"gmail_signature">-- <br>J=C3=A9r=C3=B4me Mainaud<br><a href=
=3D"mailto:jerome@mainaud.com" target=3D"_blank">jerome@mainaud.com</a><br>=
</div></div>
</div></div></div></font></span></div></div></div></div></div></div>
</blockquote></div><br></div>

--f403043612f0cbc3290545e55ed7--