From user-return-25963-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon May 7 10:59:52 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 10EC1C8A4 for ; Mon, 7 May 2012 10:59:52 +0000 (UTC) Received: (qmail 99812 invoked by uid 500); 7 May 2012 10:59:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 99787 invoked by uid 500); 7 May 2012 10:59:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 99779 invoked by uid 99); 7 May 2012 10:59:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 May 2012 10:59:49 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a59.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 May 2012 10:59:42 +0000 Received: from homiemail-a59.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a59.g.dreamhost.com (Postfix) with ESMTP id AD71656405C for ; Mon, 7 May 2012 03:59:20 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=MYOBjgrurn UbKVbc9rT9SF4lr0AAzDo4NqKitwvKYJBf8ej4YP5r6DpNUS91u8/MaEEI8qqCrx 3FeEiBFAaJSx8JD4JlQSHpCCDf3+bMOj/V/gGbaL72P8rxkrQVY8K0goxc+QVl0E 9J3LUH9xBFIiqm4QdxOheHUxRjkHpVLAY= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=tJxGXeIatNpiaN3q UCeBu0uvRnI=; b=y2/u8r3CsyaY9gteyrbJ5Y3y4Bf5DcUKBoiyQ3m45/CNxgTZ Z3MUM16QcLwCYUu6Hs89jgNSKtRyKArVOVdq3osy4Jx3mbEyUPP5TLJjCo8ne835 cfhdoNTPjsugtZrVQerja4gKOMlq9A1h8DPga+m3sboflzrSiRoxjVYvCTA= Received: from [172.16.1.4] (253.194.69.111.dynamic.snap.net.nz [111.69.194.253]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a59.g.dreamhost.com (Postfix) with ESMTPSA id 2AA12564059 for ; Mon, 7 May 2012 03:59:20 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: multipart/alternative; boundary="Apple-Mail=_6C170D95-BCC8-4173-B3E0-53A69C85FCE2" Subject: Re: count after truncate NOT zero Date: Mon, 7 May 2012 22:59:16 +1200 In-Reply-To: <4FA2880D.5020300@adyen.com> To: user@cassandra.apache.org References: <4FA2880D.5020300@adyen.com> Message-Id: X-Mailer: Apple Mail (2.1257) --Apple-Mail=_6C170D95-BCC8-4173-B3E0-53A69C85FCE2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 I don't know the YCSB code, but one theory would be=85 1) The cluster is overloaded by the test.=20 2) A write at CL ALL fails because a node does not respond in time.=20 3) The coordinator stores the hint and returns failure to the client.=20 4) The client gets an UnavailableException and retries the operation.=20 Did the nodes show any dropped messages ? Either in nodetool tpstats or = in the logs? Truncate is meta data operation, unlike deleting columns or rows. When a = column is deleted a Tombstone column is written, when row is deleted = information is associated with the key, in the context of the CF. = Truncate snapshots and then deletes the SSTables on disk, it does not = write to the SSTables. So it is possible for a write to be stored with a = lower timestamp than the truncate, because truncate does not have a = timestamp.=20 cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 4/05/2012, at 1:28 AM, Peter Dijkshoorn wrote: > Hi guys, >=20 > I got a weird thingy popping up twice today, I run a test where I = insert > a milion records via YCSB and edited it to allow me to adjust the > consistency level: the write operations are done with = ConsistencyLevel.ALL. > This is send to a 4 (virtual) node cluster with a keyspace 'test' set = up > with replication factor 3. > Now I expect that because of the ConsistencyLevel.ALL there is no = hinted > handoff active, since writes are to be accepted by all nodes before = the > operation returns to the client. The client gets only OK back, none = fails. > After the test I run a truncate, and a count which reveals still = active > records, time does not matter, I have to re-invoke the truncate to > remove the records. >=20 > [cqlsh 2.0.0 | Cassandra 1.0.8 | CQL spec 2.0.0 | Thrift protocol = 19.20.0] > cqlsh> use test; > cqlsh:test> truncate usertable; > cqlsh:test> select count(*) from usertable ; > count > ------- > 3 >=20 >=20 > On the cassandra output (-f) I can see that there is some handoff-ing > active, which I did not expect. >=20 > Has anyone an idea why the handoff is active while issuing opperations > with ConsistencyLevel.ALL? > Why is the truncate not correctly put in sync and allows subsequent > handoff's delivered of records originally set before the truncate? >=20 > Thanks if you can clarify these thing, I did not expect this at all. >=20 > Cheers, >=20 > Peter >=20 > --=20 > Peter Dijkshoorn > Adyen - Payments Made Easy > www.adyen.com >=20 > Visiting address: Mail Address: =20 > Simon Carmiggeltstraat 6-50 P.O. Box 10095 > 1011 DJ Amsterdam 1001 EB Amsterdam > The Netherlands The Netherlands >=20 > Office +31.20.240.1240 > Email peter.dijkshoorn@adyen.com >=20 --Apple-Mail=_6C170D95-BCC8-4173-B3E0-53A69C85FCE2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 I = don't know the YCSB code, but one theory would be=85

1)= The cluster is overloaded by the test. 
2) A write at CL = ALL fails because a node does not respond in time. 
3) = The coordinator stores the hint and returns failure to the = client. 
4) The client gets an UnavailableException and = retries the operation. 

Did the nodes show = any dropped messages ? Either in nodetool tpstats or in the = logs?

Truncate is meta data operation, unlike = deleting columns or rows. When a column is deleted a Tombstone column is = written, when row is deleted information is associated with the key, in = the context of the CF. Truncate snapshots and then deletes the SSTables = on disk, it does not write to the SSTables. So it is possible for a = write to be stored with a lower timestamp than the truncate, because = truncate does not have a = timestamp. 

cheers

=
http://www.thelastpickle.com

On 4/05/2012, at 1:28 AM, Peter Dijkshoorn wrote:

Hi = guys,

I got a weird thingy popping up twice today, I run a test = where I insert
a milion records via YCSB and edited it to allow me to = adjust the
consistency level: the write operations are done with = ConsistencyLevel.ALL.
This is send to a 4 (virtual) node cluster with = a keyspace 'test' set up
with replication factor 3.
Now I expect = that because of the ConsistencyLevel.ALL there is no hinted
handoff = active, since writes are to be accepted by all nodes before = the
operation returns to the client. The client gets only OK back, = none fails.
After the test I run a truncate, and a count which = reveals still active
records, time does not matter, I have to = re-invoke the truncate to
remove the records.

[cqlsh 2.0.0 | = Cassandra 1.0.8 | CQL spec 2.0.0 | Thrift protocol 19.20.0]
cqlsh> = use test;
cqlsh:test> truncate usertable;
cqlsh:test> select = count(*) from usertable ;
count
-------
=     3


On the cassandra output (-f) I can = see that there is some handoff-ing
active, which I did not = expect.

Has anyone an idea why the handoff is active while = issuing opperations
with ConsistencyLevel.ALL?
Why is the truncate = not correctly put in sync and allows subsequent
handoff's delivered = of records originally set before the truncate?

Thanks if you can = clarify these thing, I did not expect this at = all.

Cheers,

Peter

--
Peter Dijkshoorn
Adyen = - Payments Made Easy
www.adyen.com

Visiting address: =     =        Mail Address: =             Simon Carmiggeltstraat 6-50     P.O. Box = 10095
1011 DJ Amsterdam =             &n= bsp; 1001 EB Amsterdam
The Netherlands =             &n= bsp;   The Netherlands

Office = +31.20.240.1240
Email = peter.dijkshoorn@adyen.com


= --Apple-Mail=_6C170D95-BCC8-4173-B3E0-53A69C85FCE2--