Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9E39717887 for ; Tue, 31 Mar 2015 14:59:37 +0000 (UTC) Received: (qmail 66111 invoked by uid 500); 31 Mar 2015 14:51:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 66071 invoked by uid 500); 31 Mar 2015 14:51:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 66061 invoked by uid 99); 31 Mar 2015 14:51:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Mar 2015 14:51:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ken.hancock@schange.com designates 209.85.212.172 as permitted sender) Received: from [209.85.212.172] (HELO mail-wi0-f172.google.com) (209.85.212.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Mar 2015 14:51:14 +0000 Received: by widdi4 with SMTP id di4so12136615wid.0 for ; Tue, 31 Mar 2015 07:49:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=y6NmQsD94q9nSwLDNuhmkTgfvTWfANBt1awleNUw1kk=; b=K9oE4zc2HvaW01sNTNuhGUQ1TKf2UjqAKiEvTmLB8iIWHn6rDdffukid836RDWiRGt w3iD5JJaxXxIdcnPal8JpzSFEf5MsXRjyet8ImDxO87fQ1+Fy+8tPXxvwsEKPWNgMF1k E3PRO6womnCOiDdsoEnBALA7/5Tn2ubbRnEjz7IKAgYX+12ezRhZqQc8Zp1YJdU1Cgrp 1yf4RaquwSdnZ0VIKWfhMY1kb+8g1ikl4sXSungAn04VqRM9LahGIe5o8ZaFkgh8mh+Y sxQvBz7hjLf6p8qMmuf1iazicz9cOUXssiEynDEWMOoQVIcuUqx0j7NQLF9mTFZI46kC Ylnw== X-Gm-Message-State: ALoCoQmqvxGqcrLRgQfrCTByZOYhQ8GWVKmDVRoErgoGxlo3l6jZlbcRAocrFPA1NMB5aybgeB5g X-Received: by 10.180.9.171 with SMTP id a11mr6317631wib.24.1427813363020; Tue, 31 Mar 2015 07:49:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.151.197 with HTTP; Tue, 31 Mar 2015 07:49:02 -0700 (PDT) In-Reply-To: References: From: Ken Hancock Date: Tue, 31 Mar 2015 10:49:02 -0400 Message-ID: Subject: Re: Why select returns tombstoned results? To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11c241e24f0f8a051296b45b X-Virus-Checked: Checked by ClamAV on apache.org --001a11c241e24f0f8a051296b45b Content-Type: text/plain; charset=UTF-8 Have you checked time sync across all servers? The fact that you've changed consistency levels and you're getting different results may indicate something inherently wrong with the cluster such as writes being dropped or time differences between the nodes. A brute-force approach to better understand what's going on (especially if you have an example of the wrong data being returned) is to do a sstable2json on all your tables and simply grep for an example key. On Mon, Mar 30, 2015 at 4:39 PM, Benyi Wang wrote: > Thanks for replying. > > In cqlsh, if I change to Quorum (Consistency quorum), sometime the select > return the deleted row, sometime not. > > I have two virtual data centers: service (3 nodes) and analytics(4 nodes > collocate with Hadoop data nodes).The table has 3 replicas in service and 2 > in analytics. When I wrote, I wrote into analytics using local_one. So I > guest the data may not replicated to all nodes yet. > > I will try to use strong consistency for write. > > > > On Mon, Mar 30, 2015 at 11:59 AM, Prem Yadav wrote: > >> Increase the read CL to quorum and you should get correct results. >> How many nodes do you have in the cluster and what is the replication >> factor for the keyspace? >> >> On Mon, Mar 30, 2015 at 7:41 PM, Benyi Wang >> wrote: >> >>> Create table tomb_test ( >>> guid text, >>> content text, >>> range text, >>> rank int, >>> id text, >>> cnt int >>> primary key (guid, content, range, rank) >>> ) >>> >>> Sometime I delete the rows using cassandra java driver using this query >>> >>> DELETE FROM tomb_test WHERE guid=? and content=? and range=? >>> >>> in Batch statement with UNLOGGED. CONSISTENCE_LEVEL is local_one. >>> >>> But if I run >>> >>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and >>> range='week' >>> or >>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and >>> range='week' and rank = 1 >>> >>> The result shows the deleted rows. >>> >>> If I run this select, the deleted rows are not shown >>> >>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' >>> >>> If I run delete statement in cqlsh, the deleted rows won't show up. >>> >>> How can I fix this? >>> >>> >> > -- *Ken Hancock *| System Architect, Advanced Advertising SeaChange International 50 Nagog Park Acton, Massachusetts 01720 ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hancock@schange.com | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks[image: LinkedIn] [image: SeaChange International] This e-mail and any attachments may contain information which is SeaChange International confidential. The information enclosed is intended only for the addressees herein and may not be copied or forwarded without permission from SeaChange International. --001a11c241e24f0f8a051296b45b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Have you checked time sync across all servers?=C2=A0 = The fact that you've changed consistency levels and you're getting = different results may indicate something inherently wrong with the cluster = such as writes being dropped or time differences between the nodes.

=
A brute-force approach to better understand what's going on (espe= cially if you have an example of the wrong data being returned) is to do a = sstable2json on all your tables and simply grep for an example key.

On Mon, Mar 30,= 2015 at 4:39 PM, Benyi Wang <bewang.tech@gmail.com> wro= te:
Thanks for repl= ying.=C2=A0

In cqlsh, if I change to Quorum (Consi= stency quorum), sometime the select return the deleted row, sometime not.= =C2=A0

I have two virtual data centers: service (3 node= s) and analytics(4 nodes collocate with Hadoop data nodes).The table has 3 = replicas in service and 2 in analytics. When I wrote, I wrote into analytic= s using local_one. So I guest the data may not replicated to all nodes yet.=

I will try to use strong consistency for write.



On Mon, Mar 30= , 2015 at 11:59 AM, Prem Yadav <ipremyadav@gmail.com> wro= te:
Increase the read CL= to quorum and you should get correct results.=C2=A0
How many nodes do = you have in the cluster and what is the replication factor for the keyspace= ?

On Mon, Mar 30, 2015 at 7:41 PM, Benyi Wang <bewang.tech@gmail.c= om> wrote:
Create table tomb_test (
=C2=A0 =C2=A0guid text,
=C2=A0 =C2= =A0content text,
=C2=A0 =C2=A0range text,
=C2=A0 =C2=A0= rank int,
=C2=A0 =C2=A0id text,
=C2=A0 =C2=A0cnt int
=C2=A0 =C2=A0primary key (guid, content, range, rank)
)

Sometime I delete the rows using cassandra java dri= ver using this query=C2=A0

DELETE FROM tomb_test W= HERE guid=3D? and content=3D? and range=3D?

in Bat= ch statement with UNLOGGED. CONSISTENCE_LEVEL is local_one.

<= /div>
But if I run=C2=A0

SELECT * FRO= M tomb_test WHERE guid=3D'guid-1' and content=3D'content-1'= and range=3D'week'
or
SELECT * FROM= tomb_test WHERE guid=3D'guid-1' and content=3D'content-1' = and range=3D'week' and rank =3D 1

Th= e result shows the deleted rows.=C2=A0

If I r= un this select, the deleted rows are not shown

SELECT * FROM tomb_test WHERE guid=3D'guid-1' and content=3D'c= ontent-1'=C2=A0

If I run delete statemen= t in cqlsh, the deleted rows won't show up.

Ho= w can I fix this?






--
Ken Hancock=C2=A0| System Architect, Advanced Advertising= =C2=A0
SeaChange Int= ernational=C2=A0
50 Nagog Park
Acton, Massachusetts 01720
<= a href=3D"mailto:ken.hancock@schange.com" target=3D"_blank">ken.hancock@sch= ange.com=C2=A0|=C2=A0www.schange.com=C2=A0| NASDAQ:SEAC=C2=A0
Offi= ce: +1 (978) 889-3329=C2=A0|=C2=A03D"Google=C2=A0ken.hancock@schange.com=C2=A0|= =C2=A03D"Skype:"hancockks=C2=A0|=C2=A03D"Yahoohancockks

3D"SeaChange
This e-mail and any attachments may contain information which is= SeaChange International confidential. The information enclosed is intended= only for the addressees herein and may not be copied or forwarded without = permission from SeaChange International.
--001a11c241e24f0f8a051296b45b--