Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of blanquer@rightscale.com
 designates 209.85.214.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BANLkTinE7eJ+CLhTinxhcmaw0qvFHVUf=Q@mail.gmail.com>
References: <5306E6F7-3429-41A9-AEB4-0DBE45F07638@gmail.com>
	<BANLkTinE7eJ+CLhTinxhcmaw0qvFHVUf=Q@mail.gmail.com>
Date: Thu, 23 Jun 2011 07:17:36 -0700
Message-ID: <BANLkTi=zAmx-1qMvnZyt0FzwntRjANY=8g@mail.gmail.com>
Subject: Re: Backup/Restore: Coordinating Cassandra Nodetool Snapshots with
 Amazon EBS Snapshots?
From: Josep Blanquer <blanquer@rightscale.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=bcaec555517a2a093404a661bf13

--bcaec555517a2a093404a661bf13
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Jun 23, 2011 at 5:04 AM, Peter Schuller <peter.schuller@infidyne.com
> wrote:

> > 1. Is it feasible to run directly against a Cassandra data directory
> > restored from an EBS snapshot? (as opposed to nodetool snapshots restored
> > from an EBS snapshot).
>
> Assuming EBS is not buggy, including honor write barriers, including
> the linux guest kernel etc, then yes. EBS snapshots of a single
> volumes are promised to be atomic. As such, a restore from an EBS
> snapshot should be semantically identical to recover after a power
> outage or sudden reboot of the node.
>
> I make no claims as to how well EBS snapshot atomicity is actually
> tested in practice.
>
>
EBS volume atomicity is good. We've had tons of experience since EBS came
out almost 4 years ago,  to back all kinds of things, including large DBs.
One important thing to have in mind though, is that EBS snapshots are done
at the block level, not at the filesystem level. So depending on the
filesystem you have on top of the drives you might need to tell the
filesystem to "make sure this is consistent or recoverable now". For
example, if you use the log-based XFS, you might need to do xfs_freeze,
snapshot disc/s, xfs_unfreeze. To make sure that the restored filesystem
data (and not only the low level disk blocks) is recoverable when you
restore them.

 Snapshotting volume stripes works exactly in the same way, you just have to
keep track of what position each snapshot has in the stripe, so you can
recreate the stripe back correctly.

The "freezing" of the filesystem might cause a quick/mini hickup, which is
usually not noticeable unless you have very stringent requirements in the
box (or if you have a very large stripe, and/or some sort of network issue
where the calls to amazon endpoint are very slow...and therefore you're
locking the FS a tad longer than you'd want to).

 Cheers,

Josep M.

--bcaec555517a2a093404a661bf13
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<br><div class=3D"gmail_quote">On Thu, Jun 23, 2011 at 5:04 AM, Peter Schul=
ler <span dir=3D"ltr">&lt;<a href=3D"mailto:peter.schuller@infidyne.com">pe=
ter.schuller@infidyne.com</a>&gt;</span> wrote:<br><blockquote class=3D"gma=
il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef=
t:1ex;">
&gt; 1. Is it feasible to run directly against a Cassandra data directory<b=
r>
&gt; restored from an EBS snapshot? (as opposed to nodetool snapshots resto=
red<br>
&gt; from an EBS snapshot).<br>
<br>
Assuming EBS is not buggy, including honor write barriers, including<br>
the linux guest kernel etc, then yes. EBS snapshots of a single<br>
volumes are promised to be atomic. As such, a restore from an EBS<br>
snapshot should be semantically identical to recover after a power<br>
outage or sudden reboot of the node.<br>
<br>
I make no claims as to how well EBS snapshot atomicity is actually<br>
tested in practice.<br><br></blockquote><div><br>EBS volume atomicity is go=
od. We&#39;ve had tons of experience since EBS came out almost 4 years ago,=
=A0 to back all kinds of things, including large DBs. One important thing t=
o have in mind though, is that EBS snapshots are done at the block level, n=
ot at the filesystem level. So depending on the filesystem you have on top =
of the drives you might need to tell the filesystem to &quot;make sure this=
 is consistent or recoverable now&quot;. For example, if you use the log-ba=
sed XFS, you might need to do xfs_freeze, snapshot disc/s, xfs_unfreeze. To=
 make sure that the restored filesystem data (and not only the low level di=
sk blocks) is recoverable when you restore them. <br>
<br>=A0Snapshotting volume stripes works exactly in the same way, you just =
have to keep track of what position each snapshot has in the stripe, so you=
 can recreate the stripe back correctly. <br><br>The &quot;freezing&quot; o=
f the filesystem might cause a quick/mini hickup, which is usually not noti=
ceable unless you have very stringent requirements in the box (or if you ha=
ve a very large stripe, and/or some sort of network issue where the calls t=
o amazon endpoint are very slow...and therefore you&#39;re locking the FS a=
 tad longer than you&#39;d want to).<br>
<br>=A0Cheers,<br><br>Josep M.<br></div></div>

--bcaec555517a2a093404a661bf13--