Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 76711FF5C for ; Thu, 4 Apr 2013 01:36:18 +0000 (UTC) Received: (qmail 81755 invoked by uid 500); 4 Apr 2013 01:36:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 81697 invoked by uid 500); 4 Apr 2013 01:36:15 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 81652 invoked by uid 99); 4 Apr 2013 01:36:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Apr 2013 01:36:15 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a58.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Apr 2013 01:36:09 +0000 Received: from homiemail-a58.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a58.g.dreamhost.com (Postfix) with ESMTP id E5DC77D805B for ; Wed, 3 Apr 2013 18:35:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=rJ8A04l1S7BzQ0jiuePBCbRCkZ A=; b=Ys3Fco+Ch5VsDSIf611eaLzypEHINbG1xTvDWlaUqG1PwQqlKAHEi7B3jp 8BvoLuz0kRFulHPAvOauWIzWEJlCv52GsxlFv/PkONvOy2RjPX0Uy76qkCi7ZGUI KDMnEMo7vxLGZePwF8Xofkf3ORjWKZBpOdonDi1GAUQRujENE= Received: from [172.20.2.191] (unknown [115.112.62.228]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a58.g.dreamhost.com (Postfix) with ESMTPSA id 503B47D8058 for ; Wed, 3 Apr 2013 18:35:58 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_8099798C-C8DA-47DF-8FEE-CEF6A924EB3C" Message-Id: <6D6E54DA-C479-4182-91F9-73861769CD20@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Repair does not fix inconsistency Date: Thu, 4 Apr 2013 07:05:41 +0530 References: <515C18C1.4000103@opera.com> To: user@cassandra.apache.org In-Reply-To: <515C18C1.4000103@opera.com> X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_8099798C-C8DA-47DF-8FEE-CEF6A924EB3C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii What version are you on ?=20 Can you run a repair on the CF and check: Does the repair detect differences in the CF and stream changes ?=20 After the streaming does it run a secondary index rebuild on the new = sstable ? (Should be in the logs) Can you provide the full query trace ?=20 Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 3/04/2013, at 5:25 PM, Michal Michalski wrote: > Hi, >=20 > TL;DR: I have inconsistend data (1 live row on node A & 1 tombstoned = row on node B) that do not get fixed by repair. What can be a problem? >=20 > Long version: >=20 > I have a CF containing Users' info, which I sometimes query by key, = and sometimes by indexed columns like email. I'm using RF=3D2. I write = with CL.ONE, but this CF is very rarely updated, so C* has a looot of = time to fix inconsistencies that may occur, so I'm fine with this (at = least in theory ;-) ). >=20 > To be clear: > - I've run a successfull cluster-wide repair on this CF before = testing, so I do not expect any inconsistency > - All indexes are built, I've rebuilt them manually before testing, so = I expect them to work properly (I mention it because it seems to be = somehow related to indexes, but I'm not sure - see below) >=20 > The problem is: >=20 > When I query (cqlsh) some rows by key (CL is default =3D ONE) I = _always_ get a correct result. However, when I query it by indexed = column, it returns nothing. >=20 > When tracing a query with CL.ALL in cqlsh, I get info that C* has: >=20 > Read 0 live cells and 1 tombstoned // for first replica node > Read 1 live cells and 0 tombstoned // for second replica node >=20 > When CL is ONE it's never asking second replica for data (possibly due = to DynamicSnitch scores or so), so it returns nothing. >=20 > Switching to CL >=3D TWO obviously fixes this problem for us, but it's = not the solution I'd like to use as I'd rather rely on fast read/write = requests with CL.ONE + frequent repairs, allowing some short-term = inconsistency. >=20 > Any ideas why it may happen that data are still inconsistent after = repair? Is there something I could have missed? >=20 > I'm mainly surprised that repair does not fix this inconsistency in = ANY way - either by pulling missing data to first replica _OR_ = tombstoning it on second replica. First one would be correct (delete was = made a long time ago and then the row reappeared), but both could make = sense, as both will make the data consistent. In this state it's = definitely inconsistent and I don't understand it :-) >=20 >=20 > M. --Apple-Mail=_8099798C-C8DA-47DF-8FEE-CEF6A924EB3C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii What = version are you on ? 

Can you run a repair on = the CF and check:

Does the repair detect = differences in the CF and stream changes ? 
After the = streaming does it run a secondary index rebuild on the new sstable ? = (Should be in the logs)

Can you provide the = full query trace = ? 

Cheers

http://www.thelastpickle.com

On 3/04/2013, at 5:25 PM, Michal Michalski <michalm@opera.com> = wrote:

Hi,

TL;DR: I have inconsistend data (1 live row on = node A & 1 tombstoned row on node B) that do not get fixed by = repair. What can be a problem?

Long version:

I have a CF = containing Users' info, which I sometimes query by key, and sometimes by = indexed columns like email. I'm using RF=3D2. I write with CL.ONE, but =  this CF is very rarely updated, so C* has a looot of time to fix = inconsistencies that may occur, so I'm fine with this (at least in = theory ;-) ).

To be clear:
- I've run a successfull = cluster-wide repair on this CF before testing, so I do not expect any = inconsistency
- All indexes are built, I've rebuilt them manually = before testing, so I expect them to work properly (I mention it because = it seems to be somehow related to indexes, but I'm not sure - see = below)

The problem is:

When I query (cqlsh) some rows by = key (CL is default =3D ONE) I _always_ get a correct result. =  However, when I query it by indexed column, it returns = nothing.

When tracing a query with CL.ALL in cqlsh, I get info = that C* has:

Read 0 live cells and 1 tombstoned =       // for first replica node
Read 1 = live cells and 0 tombstoned       // for = second replica node

When CL is ONE it's never asking second = replica for data (possibly due to DynamicSnitch scores or so), so it = returns nothing.

Switching to CL >=3D TWO obviously fixes this = problem for us, but it's not the solution I'd like to use as I'd rather = rely on fast read/write requests with CL.ONE + frequent repairs, = allowing some short-term inconsistency.

Any ideas why it may = happen that data are still inconsistent after repair? Is there something = I could have missed?

I'm mainly surprised that repair does not = fix this inconsistency in ANY way - either by pulling missing data to = first replica _OR_ tombstoning it on second replica. First one would be = correct (delete was made a long time ago and then the row reappeared), = but both could make sense, as both will make the data consistent. In = this state it's definitely inconsistent and I don't understand it = :-)


M.

= --Apple-Mail=_8099798C-C8DA-47DF-8FEE-CEF6A924EB3C--