Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D5189101A8 for ; Wed, 16 Oct 2013 05:13:13 +0000 (UTC) Received: (qmail 77826 invoked by uid 500); 16 Oct 2013 05:13:10 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 77724 invoked by uid 500); 16 Oct 2013 05:13:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 77715 invoked by uid 99); 16 Oct 2013 05:13:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Oct 2013 05:13:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shutyaev@gmail.com designates 209.85.217.182 as permitted sender) Received: from [209.85.217.182] (HELO mail-lb0-f182.google.com) (209.85.217.182) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Oct 2013 05:13:00 +0000 Received: by mail-lb0-f182.google.com with SMTP id p9so292473lbv.27 for ; Tue, 15 Oct 2013 22:12:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=cSQ3Do6MBCgOewCo85ixDZhTyu8TTJJqlDsEyYc/Yds=; b=nG9jl1dznoxMMZ1scm9Stax0UKuYODI4evWr434SJT4DLbRzRpYvhuYIJAM37idEky 6ltJ4j+JRSq2Rw1It6Xt9KQlVwNnJ1gJi0jr9I7FSppJjqZAj+tDLNZnyKWLpf9QgOa6 h+9GcIJUgDNboE0yuWqiBV4eOeb5i7c2UNsg3t2y2EZzKqWNdA3E34xJlGlvEOco4GjJ ol1LeKWCzI9SE+yS+Bb3HzGEpUQ282XUEF28cYI4dTZoAs0cvehntPSmKMKCMvD66ibQ bCt5VBB18UrY0nCSYm1A+O38JcQ/7W8zotYv+Du3pwyLPEb00aateL8o6lN424GP/pQE oWzA== MIME-Version: 1.0 X-Received: by 10.152.121.3 with SMTP id lg3mr803082lab.0.1381900358901; Tue, 15 Oct 2013 22:12:38 -0700 (PDT) Received: by 10.112.135.34 with HTTP; Tue, 15 Oct 2013 22:12:38 -0700 (PDT) In-Reply-To: <64F4809D-A694-4DF4-A72A-E37BE980740A@jonhaddad.com> References: <64F4809D-A694-4DF4-A72A-E37BE980740A@jonhaddad.com> Date: Wed, 16 Oct 2013 09:12:38 +0400 Message-ID: Subject: Re: DELETE does not delete :) From: Alexander Shutyaev To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e0112d16401e60304e8d4c075 X-Virus-Checked: Checked by ClamAV on apache.org --089e0112d16401e60304e8d4c075 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi all, Unfortunately, we still have a problem. I've modified my code, so that it explicitly sets the consistency level to *QUORUM* for each query. However, we found out a few cases when the record is deleted on only *1 node of 3*. In this cases the *delete* query executed ok, and the *select* query that we do right after delete returned *0* rows. Later when we ran a global daily check *select* returned *1* row. How can that be? What can we be missing? 2013/10/7 Jon Haddad > I haven't used VMWare but it seems odd that it would lock up the ntp port= . > try "ps aux | grep ntp" to see if ntpd it's already running. > > On Oct 7, 2013, at 12:23 AM, Alexander Shutyaev > wrote: > > Hi Micha=C5=82, > > I didn't notice your message at first.. Well this seems like a real cause > candidate.. I'll add an explicit consistency level QUORUM and see if that > helps. Thanks > > > 2013/10/7 Alexander Shutyaev > >> Hi Nick, >> >> Thanks for the note! We have our cassanra instances installed on virtual >> hosts in VMWare and the clock synchronization is handled by the latter, = so >> I can't use ntpdate (says that NTP socket is in use). Is there any way t= o >> check if the clocks are really synchronized? My best attempt was using >> three shell windows with commands already typed thus requiring only >> clicking on the window and hitting enter. The results varied by 100-200 >> msec which I guess is just about the time I need to click and press ente= r :) >> >> Thanks in advance, >> Alexander >> >> >> 2013/10/7 Nikolay Mihaylov >> >>> Hi >>> >>> my two cents - before doing anything else, make sure clocks are >>> synchronized to the millisecond. >>> ntp will do so. >>> >>> Nick. >>> >>> >>> On Mon, Oct 7, 2013 at 9:02 AM, Alexander Shutyaev = wrote: >>> >>>> Hi all, >>>> >>>> We have encountered the following problem with cassandra. >>>> >>>> * We use *cassandra v2.0.0* from *Datastax* community repo. >>>> >>>> * We have *3 nodes* in a cluster, all of them are seed providers. >>>> >>>> * We have a *single keyspace* with *replication factor =3D 3*: >>>> >>>> *CREATE KEYSPACE bof WITH replication =3D {* >>>> * 'class': 'SimpleStrategy',* >>>> * 'replication_factor': '3'* >>>> *};* >>>> >>>> * We use *Datastax Java CQL Driver v1.0.3* in our application. >>>> >>>> * We have not modified any *consistency settings* in our app, so I >>>> assume we have the *default QUORUM* (2 out of 3 in our case) >>>> consistency *for reads and writes*. >>>> >>>> * We have 400+ tables which can be divided in two groups (*main* and * >>>> uids*). All tables in a group have the same definition, they vary only >>>> by name. The sample definitions are: >>>> >>>> *CREATE TABLE bookingfile (* >>>> * key text,* >>>> * entity_created timestamp,* >>>> * entity_createdby text,* >>>> * entity_entitytype text,* >>>> * entity_modified timestamp,* >>>> * entity_modifiedby text,* >>>> * entity_status text,* >>>> * entity_uid text,* >>>> * entity_updatepolicy text,* >>>> * version_created timestamp,* >>>> * version_createdby text,* >>>> * version_data blob,* >>>> * version_dataformat text,* >>>> * version_datasource text,* >>>> * version_modified timestamp,* >>>> * version_modifiedby text,* >>>> * version_uid text,* >>>> * version_versionnotes text,* >>>> * version_versionnumber int,* >>>> * versionscount int,* >>>> * PRIMARY KEY (key)* >>>> *) WITH* >>>> * bloom_filter_fp_chance=3D0.010000 AND* >>>> * caching=3D'KEYS_ONLY' AND* >>>> * comment=3D'' AND* >>>> * dclocal_read_repair_chance=3D0.000000 AND* >>>> * gc_grace_seconds=3D864000 AND* >>>> * index_interval=3D128 AND* >>>> * read_repair_chance=3D0.100000 AND* >>>> * replicate_on_write=3D'true' AND* >>>> * populate_io_cache_on_flush=3D'false' AND* >>>> * default_time_to_live=3D0 AND* >>>> * speculative_retry=3D'NONE' AND* >>>> * memtable_flush_period_in_ms=3D0 AND* >>>> * compaction=3D{'class': 'SizeTieredCompactionStrategy'} AND* >>>> * compression=3D{'sstable_compression': 'LZ4Compressor'};* >>>> >>>> *CREATE TABLE bookingfile_uids (* >>>> * date text,* >>>> * timeanduid text,* >>>> * deleted boolean,* >>>> * PRIMARY KEY (date, timeanduid)* >>>> *) WITH* >>>> * bloom_filter_fp_chance=3D0.010000 AND* >>>> * caching=3D'KEYS_ONLY' AND* >>>> * comment=3D'' AND* >>>> * dclocal_read_repair_chance=3D0.000000 AND* >>>> * gc_grace_seconds=3D864000 AND* >>>> * index_interval=3D128 AND* >>>> * read_repair_chance=3D0.100000 AND* >>>> * replicate_on_write=3D'true' AND* >>>> * populate_io_cache_on_flush=3D'false' AND* >>>> * default_time_to_live=3D0 AND* >>>> * speculative_retry=3D'NONE' AND* >>>> * memtable_flush_period_in_ms=3D0 AND* >>>> * compaction=3D{'class': 'SizeTieredCompactionStrategy'} AND* >>>> * compression=3D{'sstable_compression': 'LZ4Compressor'};* >>>> * >>>> * >>>> *CREATE INDEX BookingFile_uids_deleted ON bookingfile_uids (deleted);* >>>> >>>> * We don't have any problems with the tables from the *main* group. >>>> >>>> * As for the tables from the *uids* group we have noticed that >>>> sometimes deletes from these tables do not do their job. They don't fa= il, >>>> they just do nothing. We have confirmed this by adding a select query = after >>>> deletes. Most times everything is ok and select returns 0 records. But >>>> sometimes (~5 out of 100,000) it returns the supposedly deleted row. >>>> >>>> * We have logged the ExecutionInfo objects with query tracing that are >>>> returned by Datastax's driver. Here are the details >>>> >>>> *DELETE FROM bookingfile_uids WHERE date=3DC20131006 AND >>>> timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e* >>>> >>>> *ExecutionInfo: [* >>>> *triedHosts=3D/10.10.30.23;* >>>> *queriedHost=3D/10.10.30.23;* >>>> *achievedConsistencyLevel=3Dnull;* >>>> *queryTrace=3D* >>>> * Message received from /10.10.30.23 on /10.10.30.19[Thread-56] at Sun >>>> Oct 06 19:55:57 MSK 2013* >>>> * Acquiring switchLock read lock on /10.10.30.19[MutationStage:49] at >>>> Sun Oct 06 19:55:57 MSK 2013* >>>> * Appending to commitlog on /10.10.30.19[MutationStage:49] at Sun Oct >>>> 06 19:55:57 MSK 2013* >>>> * Adding to bookingfile_uids memtable on /10.10.30.19[MutationStage:49= ] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Enqueuing response to /10.10.30.23 on /10.10.30.19[MutationStage:49] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Sending message to /10.10.30.23 on /10.10.30.19[WRITE-/10.10.30.23] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Message received from /10.10.30.23 on /10.10.30.20[Thread-34] at Sun >>>> Oct 06 19:55:57 MSK 2013* >>>> * Acquiring switchLock read lock on /10.10.30.20[MutationStage:43] at >>>> Sun Oct 06 19:55:57 MSK 2013* >>>> * Appending to commitlog on /10.10.30.20[MutationStage:43] at Sun Oct >>>> 06 19:55:57 MSK 2013* >>>> * Adding to bookingfile_uids memtable on /10.10.30.20[MutationStage:43= ] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Enqueuing response to /10.10.30.23 on /10.10.30.20[MutationStage:43] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Sending message to /10.10.30.23 on /10.10.30.20[WRITE-/10.10.30.23] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Determining replicas for mutation on /10.10.30.23[Native-Transport-R= equests:1387368] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Sending message to /10.10.30.19 on /10.10.30.23[WRITE-/10.10.30.19] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Acquiring switchLock read lock on /10.10.30.23[MutationStage:46] at >>>> Sun Oct 06 19:55:57 MSK 2013* >>>> * Sending message to /10.10.30.20 on /10.10.30.23[WRITE-/10.10.30.20] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Message received from /10.10.30.20 on /10.10.30.23[Thread-5] at Sun >>>> Oct 06 19:55:57 MSK 2013* >>>> * Processing response from /10.10.30.20 on /10.10.30.23[RequestRespons= eStage:4] >>>> at Sun Oct 06 19:55:57 MSK 2013* >>>> * Message received from /10.10.30.19 on /10.10.30.23[Thread-7] at Sun >>>> Oct 06 19:55:57 MSK 2013* >>>> * Processing response from /10.10.30.19 on /10.10.30.23[RequestRespons= eStage:4] >>>> at Sun Oct 06 19:55:57 MSK 2013;* >>>> *]* >>>> >>>> *SELECT * FROM bookingfile_uids WHERE date=3DC20131006 AND >>>> timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e returned 1= record >>>> * >>>> >>>> the same query 1 second later: >>>> >>>> *DELETE FROM bookingfile_uids WHERE date=3DC20131006 AND >>>> timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e* >>>> * >>>> * >>>> *ExecutionInfo: [* >>>> *triedHosts=3D/10.10.30.20;* >>>> *queriedHost=3D/10.10.30.20;* >>>> *achievedConsistencyLevel=3Dnull;* >>>> *queryTrace=3D* >>>> * Message received from /10.10.30.20 on /10.10.30.19[Thread-57] at Sun >>>> Oct 06 19:55:58 MSK 2013* >>>> * Determining replicas for mutation on /10.10.30.20[Native-Transport-R= equests:1387705] >>>> at Sun Oct 06 19:55:58 MSK 2013* >>>> * Acquiring switchLock read lock on /10.10.30.20[MutationStage:43] at >>>> Sun Oct 06 19:55:58 MSK 2013* >>>> * Appending to commitlog on /10.10.30.20[MutationStage:43] at Sun Oct >>>> 06 19:55:58 MSK 2013* >>>> * Adding to bookingfile_uids memtable on /10.10.30.20[MutationStage:43= ] >>>> at Sun Oct 06 19:55:58 MSK 2013* >>>> * Sending message to /10.10.30.19 on /10.10.30.20[WRITE-/10.10.30.19] >>>> at Sun Oct 06 19:55:58 MSK 2013* >>>> * Sending message to /10.10.30.23 on /10.10.30.20[WRITE-/10.10.30.23] >>>> at Sun Oct 06 19:55:58 MSK 2013* >>>> * Message received from /10.10.30.19 on /10.10.30.20[Thread-4] at Sun >>>> Oct 06 19:55:58 MSK 2013* >>>> * Processing response from /10.10.30.19 on /10.10.30.20[RequestRespons= eStage:6] >>>> at Sun Oct 06 19:55:58 MSK 2013* >>>> * Message received from /10.10.30.20 on /10.10.30.23[Thread-18] at Sun >>>> Oct 06 19:55:58 MSK 2013* >>>> *]* >>>> * >>>> * >>>> *SELECT * FROM bookingfile_uids WHERE date=3DC20131006 AND >>>> timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e returned 0 >>>> records.* >>>> >>>> * Cassandra's system.log on all 3 nodes lists nothing special during >>>> these queries - just some compaction related *INFO* entries. >>>> >>>> Can anyone help with this? What is our next step? >>>> >>>> Thanks in advance, >>>> Alexander >>>> >>> >>> >> > > --089e0112d16401e60304e8d4c075 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi all,

Unfortunately, we still h= ave a problem. I've modified my code, so that it explicitly sets the co= nsistency level to QUORUM for each query. However, we found out a fe= w cases when the record is deleted on only 1 node of 3. In this case= s the delete query executed ok, and the select query that we = do right after delete returned 0 rows. Later when we ran a global da= ily check select returned 1 row. How can that be? What can we= be missing?


2013/10= /7 Jon Haddad <jon@jonhaddad.com>
I haven't used VMWare but it s= eems odd that it would lock up the ntp port. =C2=A0try "ps aux | grep = ntp" to see if ntpd it's already running.

On Oct 7, 2013, at 12:23 AM, Alexander Shutyaev <shutyaev@gmail.com> wrote= :

Hi Micha=C5=82,

I didn't notic= e your message at first.. Well this seems like a real cause candidate.. I&#= 39;ll add an explicit consistency level QUORUM and see if that helps. Thank= s


2013/10= /7 Alexander Shutyaev <shutyaev@gmail.com>
Hi Nick,

Thanks for the note! We = have our cassanra instances installed on virtual hosts in VMWare and the cl= ock synchronization is handled by the latter, so I can't use ntpdate (s= ays that NTP socket is in use). Is there any way to check if the clocks are= really synchronized? My best attempt was using three shell windows with co= mmands already typed thus requiring only clicking on the window and hitting= enter. The results varied by 100-200 msec which I guess is just about the = time I need to click and press enter :)

Thanks in advance,
Alexander
=


2013/10/7 Nik= olay Mihaylov <nmmm@nmmm.nu>
Hi

my tw= o cents - before doing anything else, make sure clocks are synchronized to = the millisecond.
ntp will do so.

Nick.


On Mon, Oct 7, 2013 at 9:02 AM, Alexande= r Shutyaev <shutyaev@gmail.com> wrote:
Hi all,

We have encountered the followi= ng problem with cassandra.

* We use cassandra v= 2.0.0 from Datastax community repo.

* We have 3 nodes in a cluster, all of them are seed = providers.

* We have a single keyspace with= replication factor =3D 3:

CREATE KEYSPACE bof WITH replication =3D {
= =C2=A0 'class': 'SimpleStrategy',
=C2=A0 &= #39;replication_factor': '3'
};
=
* We use Datastax Java CQL Driver v1.0.3 in our applicati= on.

* We have not modified any consistency sett= ings in our app, so I assume we have the default QUORUM (2 out o= f 3 in our case) consistency for reads and writes.

* We have 400+ tables which can be divided in two group= s (main and uids). All tables in a group have the same defini= tion, they vary only by name. The sample definitions are:

CREATE TABLE bookingfile (
=C2= =A0 key text,
=C2=A0 entity_created timestamp,
=C2=A0 entity_createdby text,
=C2=A0 entity_entityty= pe text,
=C2=A0 entity_modified timestamp,
=C2=A0 entity_mo= difiedby text,
=C2=A0 entity_status text,
=C2=A0 entity_uid text,
=C2=A0 entity_updatepolicy text,<= /b>
=C2=A0 version_created timestamp,
=C2=A0 version_createdby text,
=C2=A0 version_data= blob,
=C2=A0 version_dataformat text,
= =C2=A0 version_datasource text,
=C2=A0 version_modified ti= mestamp,
=C2=A0 version_modifiedby text,
=C2=A0 version_uid text= ,
=C2=A0 version_versionnotes text,
=C2= =A0 version_versionnumber int,
=C2=A0 versionscount int,
=C2=A0 PRIMARY KEY (key)
) WITH
=C2=A0 bloom_filter_fp_chance=3D0.010000 AN= D
=C2=A0 caching=3D'KEYS_ONLY' AND
<= b>=C2=A0 comment=3D'' AND
=C2=A0 dclocal_read_repa= ir_chance=3D0.000000 AND
=C2=A0 gc_grace_seconds=3D864000 AND
=C2=A0 index_= interval=3D128 AND
=C2=A0 read_repair_chance=3D0.100000 AN= D
=C2=A0 replicate_on_write=3D'true' AND
=
=C2=A0 populate_io_cache_on_flush=3D'false' AND
=C2=A0 default_time_to_live=3D0 AND
=C2=A0 specula= tive_retry=3D'NONE' AND
=C2=A0 memtable_flush_peri= od_in_ms=3D0 AND
=C2=A0 compaction=3D{'class': = 9;SizeTieredCompactionStrategy'} AND
=C2=A0 compression=3D{'sstable_compression': 'LZ4Compre= ssor'};

CREATE TABLE bookingfile_u= ids (
=C2=A0 date text,
=C2=A0 timeanduid= text,
=C2=A0 deleted boolean,
=C2=A0 PRIMARY KEY (date, = timeanduid)
) WITH
=C2=A0 bloom_filter_fp= _chance=3D0.010000 AND
=C2=A0 caching=3D'KEYS_ONLY'= ; AND
=C2=A0 comment=3D'' AND
=C2=A0 dclocal_read_rep= air_chance=3D0.000000 AND
=C2=A0 gc_grace_seconds=3D864000= AND
=C2=A0 index_interval=3D128 AND
=C2= =A0 read_repair_chance=3D0.100000 AND
=C2=A0 replicate_on_write=3D'true' AND
=C2= =A0 populate_io_cache_on_flush=3D'false' AND
=C2= =A0 default_time_to_live=3D0 AND
=C2=A0 speculative_retry= =3D'NONE' AND
=C2=A0 memtable_flush_period_in_ms=3D0 AND
=C2=A0 = compaction=3D{'class': 'SizeTieredCompactionStrategy'} AND<= /b>
=C2=A0 compression=3D{'sstable_compression': '= LZ4Compressor'};

CREATE INDEX BookingFile_uids_deleted ON book= ingfile_uids (deleted);

* We don't h= ave any problems with the tables from the main group.

* As for the tables from the uids=C2=A0group we = have noticed that sometimes deletes from these tables do not do their job. = They don't fail, they just do nothing. We have confirmed this by adding= a select query after deletes. Most times everything is ok and select retur= ns 0 records. But sometimes (~5 out of 100,000) it returns the supposedly d= eleted row.

* We have logged the ExecutionInfo objects with query t= racing that are returned by Datastax's driver. Here are the details

DELETE FROM bookingfile_uids WHERE date=3DC2013100= 6 AND timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e

ExecutionInfo: [
triedHosts=3D= /10.10.30.23;
queriedHost=3D/= 10.10.30.23;
achievedConsistencyLevel=3Dnull;
queryTrace=3D
Message received from= /10.10.30.23 on /10.10.30.19[Thread-56] a= t Sun Oct 06 19:55:57 MSK 2013
Appending to commitlog on /10.10.30.19[MutationStage:49= ] at Sun Oct 06 19:55:57 MSK 2013
Adding to bookingfile_= uids memtable on /10.10.3= 0.19[MutationStage:49] at Sun Oct 06 19:55:57 MSK 2013
Enqueuing response to /10.10.30.23 on /10.10.30.19[MutationStage:49] at= Sun Oct 06 19:55:57 MSK 2013
Sending message to /10.10.30.23 on /10.10.30.19[WRITE-/10.10.30.23] at Sun Oct 06 1= 9:55:57 MSK 2013
Message received from = /10.10.30.23 on /10.10.30.20[Thread-34] at= Sun Oct 06 19:55:57 MSK 2013
Appending to commitlog on /10.10.30.20[MutationStage:43= ] at Sun Oct 06 19:55:57 MSK 2013
Adding to bookingfile_= uids memtable on /10.10.3= 0.20[MutationStage:43] at Sun Oct 06 19:55:57 MSK 2013
Enqueuing response to /10.10.30.23 on /10.10.30.20[MutationStage:43] at= Sun Oct 06 19:55:57 MSK 2013
Sending message to /10.10.30.23 on /10.10.30.20[WRITE-/10.10.30.23] at Sun Oct 06 1= 9:55:57 MSK 2013
Determining replicas f= or mutation on /10.10.30.= 23[Native-Transport-Requests:1387368] at Sun Oct 06 19:55:57 MSK 2013
Sending message to /10.10.30.19 on /10.10.30.23[WRITE-/10.10.30.19] at Sun Oct 06 1= 9:55:57 MSK 2013
Sending message to /10.10.30.20 on /10.10.30.23[WRITE-/10.10.30.20] at Sun Oct 06 19:55:57 MSK 20= 13
Message received from = /10.10.30.20 on /10.10.30.23[Thread-5] at = Sun Oct 06 19:55:57 MSK 2013
Processing response fr= om /10.10.30.20 on /<= a href=3D"http://10.10.30.23/" target=3D"_blank">10.10.30.23[RequestRes= ponseStage:4] at Sun Oct 06 19:55:57 MSK 2013
Message received from = /10.10.30.19 on /10.10.30.23[Thread-7] at = Sun Oct 06 19:55:57 MSK 2013
Processing response fr= om /10.10.30.19 on /<= a href=3D"http://10.10.30.23/" target=3D"_blank">10.10.30.23[RequestRes= ponseStage:4] at Sun Oct 06 19:55:57 MSK 2013;
]

SELECT * FROM bookingfile_uids WH= ERE date=3DC20131006 AND timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7f= cb6b7394e returned 1 record

the same query= 1 second later:

DELETE FROM bookingfile_uids WHERE date=3DC2013= 1006 AND timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e

ExecutionInfo: [
triedHosts=3D/10.10.30= .20;
queriedHost=3D/10.10.30.20;
achievedConsistencyLevel= =3Dnull;
queryTrace=3D
Message received from = /10.10.30.20 on /10.10.30.19[Thread-57] at= Sun Oct 06 19:55:58 MSK 2013
Determining replicas f= or mutation on /10.10.30.= 20[Native-Transport-Requests:1387705] at Sun Oct 06 19:55:58 MSK 2013
Appending to commitlog= on /10.10.30.20[Muta= tionStage:43] at Sun Oct 06 19:55:58 MSK 2013
Adding to bookingfile_uids memtable on /<= a href=3D"http://10.10.30.20/" target=3D"_blank">10.10.30.20[MutationSt= age:43] at Sun Oct 06 19:55:58 MSK 2013
Sending message to /10.10.30.19 on /10.10.30.20[WRITE-/10.10.30.19] at Sun Oct 06 1= 9:55:58 MSK 2013
Sending message to /10.10.30.23 on /10.10.30.20[WRITE-/10.10.30.23] at Sun Oct 06 1= 9:55:58 MSK 2013
Message received from = /10.10.30.19 on /10.10.30.20[Thread-4] at = Sun Oct 06 19:55:58 MSK 2013
Processing response fr= om /10.10.30.19 on /<= a href=3D"http://10.10.30.20/" target=3D"_blank">10.10.30.20[RequestRes= ponseStage:6] at Sun Oct 06 19:55:58 MSK 2013
Message received from = /10.10.30.20 on /10.10.30.23[Thread-18] at= Sun Oct 06 19:55:58 MSK 2013
]

SELECT * FROM bookingfile_= uids WHERE date=3DC20131006 AND timeAndUid=3D195248590_4762ce41-d2d2-448d-b= e8c-c7fcb6b7394e returned 0 records.

* Cassand= ra's system.log on all 3 nodes lists nothing special during these queri= es - just some compaction related INFO entries.

Can anyone help with this? What is our next step?
=

Thanks in advance,
Alexander





--089e0112d16401e60304e8d4c075--