Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of shaun@cuttshome.net
 designates 69.17.117.50 as permitted sender)
From: Shaun Cutts <shaun@cuttshome.net>
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: multipart/alternative; boundary=Apple-Mail-73-1041093609
Subject: Re: Best way to detect/fix bitrot today?
Date: Tue, 8 Feb 2011 09:37:53 -0500
In-Reply-To: <AANLkTinCfhjq7HYNcsBqqhhY1PrQsE4SMOFoE49kACHm@mail.gmail.com>
To: user@cassandra.apache.org
References: <AANLkTimV5oEw6Vm5OKRXnwudY3csye4xsk87L_FtWWm5@mail.gmail.com>
 <AANLkTi=aUqcY3uQ0FP92bJFS=vAUa0QQVx6W3sbAZV+G@mail.gmail.com>
 <AANLkTinCfhjq7HYNcsBqqhhY1PrQsE4SMOFoE49kACHm@mail.gmail.com>
Message-Id: <F8CAF15D-C38F-473E-84B9-C38783E2FBBB@cuttshome.net>


--Apple-Mail-73-1041093609
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

One thing that we're doing for (guaranteed) immutable data is to use MD5 =
signatures as keys... this will also prevent duplication, and it will =
allow detection (if not correction) of bitrot at the app level easy.

On Feb 8, 2011, at 9:23 AM, Anand Somani wrote:

> I should have clarified we have 3 copies, so in that case as long as 2 =
match we should be ok?=20
>=20
> Even if there were checksumming at the SStable level, I assume it has =
to check and report these errors on compaction (or node repair)?=20
>=20
> I have seen some JIRA open on these issues ( 47 and 1717), but if I =
need something today, a read repair ( or a node repair) is the only =
viable option?=20
>=20
> =20
>=20
> On Mon, Feb 7, 2011 at 12:09 PM, Peter Schuller =
<peter.schuller@infidyne.com> wrote:
> > Our application space is such that there is data that might not be =
read for
> > a long time. The data is mostly immutable. How should I approach
> > detecting/solving the bitrot problem? One approach is read data and =
let read
> > repair do the detection, but given the size of data, that does not =
look very
> > efficient.
>=20
> Note that read-repair is not really intended to repair arbitrary
> corruptions. Unless I'm mistaken, arbitrary corruption, unless it
> triggers a serialization failure that causes row skipping, it's a
> toss-up which version of the data is retained (or both, if the
> corruption is in the key). Given the same key and column timestamp,
> the tie breaker is the volumn value. So depending on whether
> corruption results in a "lesser" or "greater" value, you might get the
> corrupt or non-corrupt data.
>=20
> > Has anybody solved/workaround this or has any other suggestions to =
detect
> > and fix bitrot?
>=20
> My feel/tentative opinion is that the clean fix is for Cassandra to
> support strong checksumming at the sstable level.
>=20
> Deploying on e.g. ZFS would help a lot with this, but that's a problem
> for deployment on Linux (which is the recommended platform for
> Cassandra).
>=20
> --
> / Peter Schuller
>=20


--Apple-Mail-73-1041093609
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">One thing that we're doing for (guaranteed) immutable data is to use MD5 signatures as keys... this will also prevent duplication, and it will allow detection (if not correction) of bitrot at the app level easy.<br>
<br><div><div>On Feb 8, 2011, at 9:23 AM, Anand Somani wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">I should have clarified we have 3 copies, so in that case as long as 2 match we should be ok? <br><br>Even if there were checksumming at the SStable level, I assume it has to check and report these errors on compaction (or node repair)? <br>

<br>I have seen some JIRA open on these issues ( 47 and 1717), but if I need something today, a read repair ( or a node repair) is the only viable option? <br><br>&nbsp;<br><br><div class="gmail_quote">On Mon, Feb 7, 2011 at 12:09 PM, Peter Schuller <span dir="ltr">&lt;<a href="mailto:peter.schuller@infidyne.com" target="_blank">peter.schuller@infidyne.com</a>&gt;</span> wrote:<br>


<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div>&gt; Our application space is such that there is data that might not be read for<br>

&gt; a long time. The data is mostly immutable. How should I approach<br>
&gt; detecting/solving the bitrot problem? One approach is read data and let read<br>
&gt; repair do the detection, but given the size of data, that does not look very<br>
&gt; efficient.<br>
<br>
</div>Note that read-repair is not really intended to repair arbitrary<br>
corruptions. Unless I'm mistaken, arbitrary corruption, unless it<br>
triggers a serialization failure that causes row skipping, it's a<br>
toss-up which version of the data is retained (or both, if the<br>
corruption is in the key). Given the same key and column timestamp,<br>
the tie breaker is the volumn value. So depending on whether<br>
corruption results in a "lesser" or "greater" value, you might get the<br>
corrupt or non-corrupt data.<br>
<div><br>
&gt; Has anybody solved/workaround this or has any other suggestions to detect<br>
&gt; and fix bitrot?<br>
<br>
</div>My feel/tentative opinion is that the clean fix is for Cassandra to<br>
support strong checksumming at the sstable level.<br>
<br>
Deploying on e.g. ZFS would help a lot with this, but that's a problem<br>
for deployment on Linux (which is the recommended platform for<br>
Cassandra).<br>
<br>
--<br>
<font color="#888888">/ Peter Schuller<br>
</font></blockquote></div><br>
</blockquote></div><br></body></html>
--Apple-Mail-73-1041093609--