Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E099498E2 for ; Wed, 28 Dec 2011 11:53:40 +0000 (UTC) Received: (qmail 1299 invoked by uid 500); 28 Dec 2011 11:53:38 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 1238 invoked by uid 500); 28 Dec 2011 11:53:37 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 1230 invoked by uid 99); 28 Dec 2011 11:53:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Dec 2011 11:53:37 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dwilliams@system7.co.uk designates 209.85.210.172 as permitted sender) Received: from [209.85.210.172] (HELO mail-iy0-f172.google.com) (209.85.210.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Dec 2011 11:53:32 +0000 Received: by iaen33 with SMTP id n33so15041629iae.31 for ; Wed, 28 Dec 2011 03:53:11 -0800 (PST) Received: by 10.50.190.196 with SMTP id gs4mr35855693igc.14.1325073191100; Wed, 28 Dec 2011 03:53:11 -0800 (PST) MIME-Version: 1.0 Received: by 10.50.135.104 with HTTP; Wed, 28 Dec 2011 03:52:50 -0800 (PST) In-Reply-To: <4EF9E145.5060001@borgstrom.se> References: <4EF9E145.5060001@borgstrom.se> From: Dominic Williams Date: Wed, 28 Dec 2011 11:52:50 +0000 Message-ID: Subject: Re: Previously deleted rows resurrected by repair? To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d04479ee3db601404b525a48a --f46d04479ee3db601404b525a48a Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hmm interesting could be some variation on 3510 (which caught me out). Personally I really don't like having to rely on repair to stop deletes being undone. If you agree follow this proposal for an alternative https://issues.apache.org/jira/browse/CASSANDRA-3620 which also stops tombstone build up. 2011/12/27 Jonas Borgstr=F6m > Hi, > > I Have a 3 node cluster running Cassandra 1.0.3 and using replication > factor=3D3. > > Recently I've noticed that some previously deleted rows have started to > reappear for some reason. And now I wonder if this is a known issue with > 1.0.3? > > Repairs have been running every weekend (gc_grace is 10 days) and always > completed successfully. But while looking at the logs I noticed that a fa= ir > number of ranges (around 10% of the total number of keys) have been > streamed between these nodes during the repair sessions. This seems a bit > high to me given that everything is written using quorum and all nodes ha= ve > been up all the time. > > For me this looks suspiciously like some already deleted keys are streame= d > to other nodes during repair. > > > Some more details about the data: > All keys are written to only once and most of them are deleted a couple o= f > days/weeks later. Some keys are large enough to require incremental > compaction. > > Could this bug cause this? > > https://issues.apache.org/**jira/browse/CASSANDRA-3510 > > Regards, > Jonas > --f46d04479ee3db601404b525a48a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hmm interesting could be some variation on 3510 (which caught me out).
=
Personally I really don't like having to rely on repair = to stop deletes being undone. If you agree follow this proposal for an alte= rnative=A0= https://issues.apache.org/jira/browse/CASSANDRA-3620=A0which also stops= tombstone build up.=A0

2011/12/27 Jonas Borgstr=F6m <jonas@borgstrom.se>
Hi,

I Have a 3 node cluster running Cassandra 1.0.3 and using replication facto= r=3D3.

Recently I've noticed that some previously deleted rows have started to= reappear for some reason. And now I wonder if this is a known issue with 1= .0.3?

Repairs have been running every weekend (gc_grace is 10 days) and always co= mpleted successfully. But while looking at the logs I noticed that a fair n= umber of ranges (around 10% of the total number of keys) have been streamed= between these nodes during the repair sessions. This seems a bit high to m= e given that everything is written using quorum and all nodes have been up = all the time.

For me this looks suspiciously like some already deleted keys are streamed = to other nodes during repair.


Some more details about the data:
All keys are written to only once and most of them are deleted a couple of = days/weeks later. Some keys are large enough to require incremental compact= ion.

Could this bug cause this?

https://issues.apache.org/jira/browse/CASSANDRA-3510

Regards,
Jonas

--f46d04479ee3db601404b525a48a--