Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6BE2211FE1 for ; Fri, 27 Jun 2014 14:25:32 +0000 (UTC) Received: (qmail 97239 invoked by uid 500); 27 Jun 2014 14:25:29 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 97199 invoked by uid 500); 27 Jun 2014 14:25:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 97189 invoked by uid 99); 27 Jun 2014 14:25:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Jun 2014 14:25:29 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of clohfink@blackbirdit.com designates 209.85.223.172 as permitted sender) Received: from [209.85.223.172] (HELO mail-ie0-f172.google.com) (209.85.223.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Jun 2014 14:25:23 +0000 Received: by mail-ie0-f172.google.com with SMTP id lx4so4439705iec.31 for ; Fri, 27 Jun 2014 07:25:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=Wztt3U+gN4HFtD6oxaIzw52rh/tOhcjr7ExO1+ULYaw=; b=mJg6Q6LvlzVOfXLkLc6gk5oiBirmwLUWcybHiUtZvGXeZVn/y+rCYuRBONbikvy6q7 SlkNARDyPn/mI5k4BYZYahrC9RFENhKaxgAzbpxlWamPsqKHkn8JOp+Tyq+H/CklQ2aI BH4ZHLOabTdKjYhgyS8IoJhNkZPsrWZMvW243lYlU+tAr5Csf928y0Lmz3tnDXLLfT56 AUMzP5ikFNPkW5Oy6sjoQj1vKc06fPBOPvonBnTu+I6I8gSPC4r7gBY3p7DA146te5/H 5LBZKyeA7nuzakPmIhCrJtf9XJluOKoM+4VJa4DDxfrBpppI3cm3qqAXASx6mFkU/kdW NfwQ== X-Gm-Message-State: ALoCoQkLobNw34b8r51eCec7ZX0w6rxzTzeHNSY+Q8Q5+vlzTtB0gJcZfWPyjClmqqd+P3zvhVvm X-Received: by 10.43.178.197 with SMTP id ox5mr20997972icc.22.1403879102078; Fri, 27 Jun 2014 07:25:02 -0700 (PDT) Received: from [192.168.0.15] (173-27-77-141.client.mchsi.com. [173.27.77.141]) by mx.google.com with ESMTPSA id d3sm14019856igc.17.2014.06.27.07.25.01 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 27 Jun 2014 07:25:01 -0700 (PDT) From: Chris Lohfink Content-Type: multipart/alternative; boundary="Apple-Mail=_1CFB231E-7414-4508-8A56-FC972F45465A" Message-Id: <6801AF02-0973-428B-AABC-A01E5A5F2429@blackbirdit.com> Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: Cassandra CLI showing inconsistent results during gets Date: Fri, 27 Jun 2014 09:25:01 -0500 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1874) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_1CFB231E-7414-4508-8A56-FC972F45465A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Where was the 09_09 column inserted from? Are you sure whatever did the = insert is doing a local_quorum on the same DC the cli is in? It may = return before all the nodes get response back (ie 2 of the 3 in local = DC) which report not having the data. After all the nodes respond it = will check the digests from all the responses, see theres an = inconsistency and do a read repair. Which would explain it showing up = following queries. Chris On Jun 26, 2014, at 10:06 AM, Ravikumar Govindarajan = wrote: > I ran the following set of commands via CLI in our servers. There is a = data-discrepancy that I encountered as below during gets... >=20 > We are running 1.2.4 version with replication-factor=3D3 (DC1) & 2 = (DC2). Reads and writes are at LOCAL_QUORUM >=20 > create column family TestCF with key_validation_class=3DAsciiType AND = comparator =3D 'CompositeType(AsciiType,LongType)' AND = compression_options=3D{sstable_compression:SnappyCompressor, = chunk_length_kb:64}; >=20 > [default@Sample] consistencylevel AS LOCAL_QUORUM; > Consistency level is set to 'LOCAL_QUORUM'. >=20 > [default@Sample] get TestCF [ascii('1773221000000008001')] = ['1773221000004550009_:1773221000004560008']; > =3D> (column=3D1773221000004550009_:1773221000004560008, = value=3D31373733323231303030303034353530303039, timestamp=3D1397743374931)= > Elapsed time: 8.64 msec(s). >=20 > //Do a full row dump which shows the above column > [default@Sample] get TestCF [ascii('1773221000000008001')]; > ........................... > =3D> (column=3D1773221000004547019_:1773221000004560001, = value=3D31373733323231303030303034353437303139, timestamp=3D1397743139121)= > =3D> (column=3D1773221000004550009_:1773221000004560008, = value=3D31373733323231303030303034353530303039, timestamp=3D1397743374931)= > =3D> (column=3D1773221000004560003_:1773221000004560005, = value=3D31373733323231303030303034353630303033, timestamp=3D1397743323261)= > =3D> (column=3D1773221000004562001_:1773221000004564003, = value=3D31373733323231303030303034353632303031, timestamp=3D1397749523707)= > --------------------------- > Returned 4771 results. > Elapsed time: 518 msec(s). > //Try again=20 > [default@Sample] get TestCF[ascii('1773221000000008001')] = ['1773221000004550009_:1773221000004560008']; > =3D> (column=3D1773221000004550009_:1773221000004560008, = value=3D31373733323231303030303034353530303039, timestamp=3D1397743374931)= > Elapsed time: 8.03 msec(s). >=20 > //Here CLI flipped showing value as not found > [default@Sample] get TestCF[ascii('1773221000000008001')] = ['1773221000004550009_:1773221000004550009']; > Value was not found > Elapsed time: 12 msec(s). >=20 > //Query again, it shows as value found > [default@Sample] get TestCF[ascii('1773221000000008001')] = ['1773221000004550009_:1773221000004550009']; > =3D> (column=3D1773221000004550009_:1773221000004550009, = value=3D31373733323231303030303034353530303039, timestamp=3D1397743374931)= > Elapsed time: 23 msec(s). >=20 > Is this just limited to CLI bug or some-thing deeper is brewing? Our = app faced a serious issue in code involving this query. Is it a known = issue? >=20 > Any help is much appreciated >=20 > -- > Ravi --Apple-Mail=_1CFB231E-7414-4508-8A56-FC972F45465A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Where = was the 09_09 column inserted from? Are you sure whatever did the insert = is doing a local_quorum on the same DC the cli is in?  It may = return before all the nodes get response back (ie 2 of the 3 in local = DC) which report not having the data.  After all the nodes respond = it will check the digests from all the responses, see theres an = inconsistency and do a read repair.  Which would explain it showing = up following queries.

Chris

On = Jun 26, 2014, at 10:06 AM, Ravikumar Govindarajan <ravikumar.govindarajan@gm= ail.com> wrote:

I ran the following set of commands via CLI in our servers. = There is a data-discrepancy that I encountered as below during = gets...

We are running 1.2.4 version with = replication-factor=3D3 (DC1) & 2 (DC2). Reads and writes are at = LOCAL_QUORUM

create column family TestCF with key_validation_class=3DAsciiType AND comparator = =3D 'CompositeType(AsciiType,LongType)' AND = compression_options=3D{sstable_compression:SnappyCompressor, = chunk_length_kb:64};

[default@Sample] consistencylevel AS = LOCAL_QUORUM;
Consistency level is set to = 'LOCAL_QUORUM'.

[default@Sample] get TestCF = [ascii('1773221000000008001')] = ['1773221000004550009_:1773221000004560008'];
=3D> (column=3D1773221000004550009_:1773221000004560008, = value=3D31373733323231303030303034353530303039, = timestamp=3D1397743374931)
Elapsed time: 8.64 = msec(s).

//Do a full row dump which shows the = above column
[default@Sample] = get TestCF [ascii('1773221000000008001')];
...........= ................
=3D> = (column=3D1773221000004547019_:1773221000004560001, = value=3D31373733323231303030303034353437303139, = timestamp=3D1397743139121)
=3D> (column=3D1773221000004550009_:1773221000004560008, = value=3D31373733323231303030303034353530303039, = timestamp=3D1397743374931)
=3D> = (column=3D1773221000004560003_:1773221000004560005, = value=3D31373733323231303030303034353630303033, = timestamp=3D1397743323261)
=3D> (column=3D1773221000004562001_:1773221000004564003, = value=3D31373733323231303030303034353632303031, = timestamp=3D1397749523707)
---------------------------
Returned 4771 results.
Elapsed time: 518 msec(s).
//Try again 
[default@Sample] = get TestCF[ascii('1773221000000008001')] ['177322100000455000= 9_:1773221000004560008'];
=3D> = (column=3D1773221000004550009_:1773221000004560008, = value=3D31373733323231303030303034353530303039, = timestamp=3D1397743374931)
Elapsed time: 8.03 msec(s).

//Here CLI = flipped showing value as not found
[default@Sample] = get TestCF[ascii('1773221000000008001')] ['177322100000455000= 9_:1773221000004550009'];
Value was not found
Elapsed time: 12 = msec(s).

//Query again, it shows as value = found
[default@Sample] = get TestCF[ascii('1773221000000008001')] ['1773221000004550009_:= 1773221000004550009'];
=3D> (column=3D1773221000004550009_:1773221000004550009, = value=3D31373733323231303030303034353530303039, = timestamp=3D1397743374931)
Elapsed time: 23 = msec(s).

Is this just limited to = CLI bug or some-thing deeper is brewing? Our app faced a serious issue = in code involving this query. Is it a known issue?

Any help is much = appreciated

--
Ravi

= --Apple-Mail=_1CFB231E-7414-4508-8A56-FC972F45465A--