Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of meatforums@gmail.com designates
 209.85.216.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hDk94KDYIiA88j2OI11rDpPQJBB1jjAK6CmXl/2kymbpxRlcex1bB49yLC9ohNIzsE
         PhtvUFhq2LmuQMRiIXMEN9XTbNaKX4kG511u6RYovQ5KVx1c7QxTZT2/n19YG9QAL3Vv
         fng171kPJhfPo1B9PFE8RsanzMyInia1scDjo=
MIME-Version: 1.0
In-Reply-To: <AANLkTi=aUqcY3uQ0FP92bJFS=vAUa0QQVx6W3sbAZV+G@mail.gmail.com>
References: <AANLkTimV5oEw6Vm5OKRXnwudY3csye4xsk87L_FtWWm5@mail.gmail.com>
	<AANLkTi=aUqcY3uQ0FP92bJFS=vAUa0QQVx6W3sbAZV+G@mail.gmail.com>
Date: Tue, 8 Feb 2011 06:23:29 -0800
Message-ID: <AANLkTinCfhjq7HYNcsBqqhhY1PrQsE4SMOFoE49kACHm@mail.gmail.com>
Subject: Re: Best way to detect/fix bitrot today?
From: Anand Somani <meatforums@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001636284566ae2cf2049bc61754

--001636284566ae2cf2049bc61754
Content-Type: text/plain; charset=ISO-8859-1

I should have clarified we have 3 copies, so in that case as long as 2 match
we should be ok?

Even if there were checksumming at the SStable level, I assume it has to
check and report these errors on compaction (or node repair)?

I have seen some JIRA open on these issues ( 47 and 1717), but if I need
something today, a read repair ( or a node repair) is the only viable
option?


On Mon, Feb 7, 2011 at 12:09 PM, Peter Schuller <peter.schuller@infidyne.com
> wrote:

> > Our application space is such that there is data that might not be read
> for
> > a long time. The data is mostly immutable. How should I approach
> > detecting/solving the bitrot problem? One approach is read data and let
> read
> > repair do the detection, but given the size of data, that does not look
> very
> > efficient.
>
> Note that read-repair is not really intended to repair arbitrary
> corruptions. Unless I'm mistaken, arbitrary corruption, unless it
> triggers a serialization failure that causes row skipping, it's a
> toss-up which version of the data is retained (or both, if the
> corruption is in the key). Given the same key and column timestamp,
> the tie breaker is the volumn value. So depending on whether
> corruption results in a "lesser" or "greater" value, you might get the
> corrupt or non-corrupt data.
>
> > Has anybody solved/workaround this or has any other suggestions to detect
> > and fix bitrot?
>
> My feel/tentative opinion is that the clean fix is for Cassandra to
> support strong checksumming at the sstable level.
>
> Deploying on e.g. ZFS would help a lot with this, but that's a problem
> for deployment on Linux (which is the recommended platform for
> Cassandra).
>
> --
> / Peter Schuller
>

--001636284566ae2cf2049bc61754
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I should have clarified we have 3 copies, so in that case as long as 2 matc=
h we should be ok? <br><br>Even if there were checksumming at the SStable l=
evel, I assume it has to check and report these errors on compaction (or no=
de repair)? <br>

<br>I have seen some JIRA open on these issues ( 47 and 1717), but if I nee=
d something today, a read repair ( or a node repair) is the only viable opt=
ion? <br><br>=A0<br><br><div class=3D"gmail_quote">On Mon, Feb 7, 2011 at 1=
2:09 PM, Peter Schuller <span dir=3D"ltr">&lt;<a href=3D"mailto:peter.schul=
ler@infidyne.com" target=3D"_blank">peter.schuller@infidyne.com</a>&gt;</sp=
an> wrote:<br>


<blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; borde=
r-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div>&gt; Our app=
lication space is such that there is data that might not be read for<br>

&gt; a long time. The data is mostly immutable. How should I approach<br>
&gt; detecting/solving the bitrot problem? One approach is read data and le=
t read<br>
&gt; repair do the detection, but given the size of data, that does not loo=
k very<br>
&gt; efficient.<br>
<br>
</div>Note that read-repair is not really intended to repair arbitrary<br>
corruptions. Unless I&#39;m mistaken, arbitrary corruption, unless it<br>
triggers a serialization failure that causes row skipping, it&#39;s a<br>
toss-up which version of the data is retained (or both, if the<br>
corruption is in the key). Given the same key and column timestamp,<br>
the tie breaker is the volumn value. So depending on whether<br>
corruption results in a &quot;lesser&quot; or &quot;greater&quot; value, yo=
u might get the<br>
corrupt or non-corrupt data.<br>
<div><br>
&gt; Has anybody solved/workaround this or has any other suggestions to det=
ect<br>
&gt; and fix bitrot?<br>
<br>
</div>My feel/tentative opinion is that the clean fix is for Cassandra to<b=
r>
support strong checksumming at the sstable level.<br>
<br>
Deploying on e.g. ZFS would help a lot with this, but that&#39;s a problem<=
br>
for deployment on Linux (which is the recommended platform for<br>
Cassandra).<br>
<br>
--<br>
<font color=3D"#888888">/ Peter Schuller<br>
</font></blockquote></div><br>

--001636284566ae2cf2049bc61754--