Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3AC051075E for ; Wed, 16 Oct 2013 15:29:16 +0000 (UTC) Received: (qmail 75755 invoked by uid 500); 16 Oct 2013 15:29:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 75503 invoked by uid 500); 16 Oct 2013 15:29:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 75494 invoked by uid 99); 16 Oct 2013 15:29:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Oct 2013 15:29:10 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.215.169] (HELO mail-ea0-f169.google.com) (209.85.215.169) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Oct 2013 15:29:03 +0000 Received: by mail-ea0-f169.google.com with SMTP id k11so452840eaj.0 for ; Wed, 16 Oct 2013 08:28:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=qvxm5BPhxzOuqjyyGOo7hI6Kp/BeDP6ElAP8bMoeueg=; b=GuA19JmiG0Xf7qj6zZ/SrnIYVmfopxIg/4XYZd6A2vcObphJxNnznSenN01d0z2+X/ ZTog/KPIWAgL+nA2PD9r/UkOfpCjpubZ1Ds/DmyeUhc7WDw/NbfzllTIKHu55ckwHPsa rfWeBgYvEHj0daHY2gFBFj9wHuls3IfOCcZoOwCuE3KQlHRFu5r8ZaLoqdJXEW4PeBK8 rr218vgkzvLQNyxcl6KK7LwrwN91YomTRKqDQqumzsFxu2gauNsc7hVjk052yIQmDRSX UOnA7grVwOalcRvU8RyDsYwNedv1mtQCRHuiMRV8V3ROq6pA5ABFU4vE/Uo95it07CbM xkeQ== X-Gm-Message-State: ALoCoQn1jHbbJYe6EHSyzig0yVipDexNbgLpoW79DRpkesVBI0ZBIJH1vKxdTvymPE5nbA2fOoHv MIME-Version: 1.0 X-Received: by 10.14.3.9 with SMTP id 9mr4075958eeg.72.1381937322811; Wed, 16 Oct 2013 08:28:42 -0700 (PDT) Received: by 10.223.63.18 with HTTP; Wed, 16 Oct 2013 08:28:42 -0700 (PDT) X-Originating-IP: [70.112.126.233] In-Reply-To: References: <64F4809D-A694-4DF4-A72A-E37BE980740A@jonhaddad.com> Date: Wed, 16 Oct 2013 10:28:42 -0500 Message-ID: Subject: Re: DELETE does not delete :) From: Nate McCall To: Cassandra Users Content-Type: multipart/alternative; boundary=047d7b6250bc3a708104e8dd5ba3 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b6250bc3a708104e8dd5ba3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable This is almost a guaranteed sign that the clocks are off in your cluster. If you run the select query a couple of times in a row right after deletion, do you see the data appear again? On Wed, Oct 16, 2013 at 12:12 AM, Alexander Shutyaev wr= ote: > Hi all, > > Unfortunately, we still have a problem. I've modified my code, so that it > explicitly sets the consistency level to *QUORUM* for each query. > However, we found out a few cases when the record is deleted on only *1 > node of 3*. In this cases the *delete* query executed ok, and the *select= *query that we do right after delete returned > *0* rows. Later when we ran a global daily check *select* returned *1*row= . How can that be? What can we be missing? > > > 2013/10/7 Jon Haddad > >> I haven't used VMWare but it seems odd that it would lock up the ntp >> port. try "ps aux | grep ntp" to see if ntpd it's already running. >> >> On Oct 7, 2013, at 12:23 AM, Alexander Shutyaev >> wrote: >> >> Hi Micha=C5=82, >> >> I didn't notice your message at first.. Well this seems like a real caus= e >> candidate.. I'll add an explicit consistency level QUORUM and see if tha= t >> helps. Thanks >> >> >> 2013/10/7 Alexander Shutyaev >> >>> Hi Nick, >>> >>> Thanks for the note! We have our cassanra instances installed on virtua= l >>> hosts in VMWare and the clock synchronization is handled by the latter,= so >>> I can't use ntpdate (says that NTP socket is in use). Is there any way = to >>> check if the clocks are really synchronized? My best attempt was using >>> three shell windows with commands already typed thus requiring only >>> clicking on the window and hitting enter. The results varied by 100-200 >>> msec which I guess is just about the time I need to click and press ent= er :) >>> >>> Thanks in advance, >>> Alexander >>> >>> >>> 2013/10/7 Nikolay Mihaylov >>> >>>> Hi >>>> >>>> my two cents - before doing anything else, make sure clocks are >>>> synchronized to the millisecond. >>>> ntp will do so. >>>> >>>> Nick. >>>> >>>> >>>> On Mon, Oct 7, 2013 at 9:02 AM, Alexander Shutyaev wrote: >>>> >>>>> Hi all, >>>>> >>>>> We have encountered the following problem with cassandra. >>>>> >>>>> * We use *cassandra v2.0.0* from *Datastax* community repo. >>>>> >>>>> * We have *3 nodes* in a cluster, all of them are seed providers. >>>>> >>>>> * We have a *single keyspace* with *replication factor =3D 3*: >>>>> >>>>> *CREATE KEYSPACE bof WITH replication =3D {* >>>>> * 'class': 'SimpleStrategy',* >>>>> * 'replication_factor': '3'* >>>>> *};* >>>>> >>>>> * We use *Datastax Java CQL Driver v1.0.3* in our application. >>>>> >>>>> * We have not modified any *consistency settings* in our app, so I >>>>> assume we have the *default QUORUM* (2 out of 3 in our case) >>>>> consistency *for reads and writes*. >>>>> >>>>> * We have 400+ tables which can be divided in two groups (*main* and = * >>>>> uids*). All tables in a group have the same definition, they vary >>>>> only by name. The sample definitions are: >>>>> >>>>> *CREATE TABLE bookingfile (* >>>>> * key text,* >>>>> * entity_created timestamp,* >>>>> * entity_createdby text,* >>>>> * entity_entitytype text,* >>>>> * entity_modified timestamp,* >>>>> * entity_modifiedby text,* >>>>> * entity_status text,* >>>>> * entity_uid text,* >>>>> * entity_updatepolicy text,* >>>>> * version_created timestamp,* >>>>> * version_createdby text,* >>>>> * version_data blob,* >>>>> * version_dataformat text,* >>>>> * version_datasource text,* >>>>> * version_modified timestamp,* >>>>> * version_modifiedby text,* >>>>> * version_uid text,* >>>>> * version_versionnotes text,* >>>>> * version_versionnumber int,* >>>>> * versionscount int,* >>>>> * PRIMARY KEY (key)* >>>>> *) WITH* >>>>> * bloom_filter_fp_chance=3D0.010000 AND* >>>>> * caching=3D'KEYS_ONLY' AND* >>>>> * comment=3D'' AND* >>>>> * dclocal_read_repair_chance=3D0.000000 AND* >>>>> * gc_grace_seconds=3D864000 AND* >>>>> * index_interval=3D128 AND* >>>>> * read_repair_chance=3D0.100000 AND* >>>>> * replicate_on_write=3D'true' AND* >>>>> * populate_io_cache_on_flush=3D'false' AND* >>>>> * default_time_to_live=3D0 AND* >>>>> * speculative_retry=3D'NONE' AND* >>>>> * memtable_flush_period_in_ms=3D0 AND* >>>>> * compaction=3D{'class': 'SizeTieredCompactionStrategy'} AND* >>>>> * compression=3D{'sstable_compression': 'LZ4Compressor'};* >>>>> >>>>> *CREATE TABLE bookingfile_uids (* >>>>> * date text,* >>>>> * timeanduid text,* >>>>> * deleted boolean,* >>>>> * PRIMARY KEY (date, timeanduid)* >>>>> *) WITH* >>>>> * bloom_filter_fp_chance=3D0.010000 AND* >>>>> * caching=3D'KEYS_ONLY' AND* >>>>> * comment=3D'' AND* >>>>> * dclocal_read_repair_chance=3D0.000000 AND* >>>>> * gc_grace_seconds=3D864000 AND* >>>>> * index_interval=3D128 AND* >>>>> * read_repair_chance=3D0.100000 AND* >>>>> * replicate_on_write=3D'true' AND* >>>>> * populate_io_cache_on_flush=3D'false' AND* >>>>> * default_time_to_live=3D0 AND* >>>>> * speculative_retry=3D'NONE' AND* >>>>> * memtable_flush_period_in_ms=3D0 AND* >>>>> * compaction=3D{'class': 'SizeTieredCompactionStrategy'} AND* >>>>> * compression=3D{'sstable_compression': 'LZ4Compressor'};* >>>>> * >>>>> * >>>>> *CREATE INDEX BookingFile_uids_deleted ON bookingfile_uids (deleted);= * >>>>> >>>>> * We don't have any problems with the tables from the *main* group. >>>>> >>>>> * As for the tables from the *uids* group we have noticed that >>>>> sometimes deletes from these tables do not do their job. They don't f= ail, >>>>> they just do nothing. We have confirmed this by adding a select query= after >>>>> deletes. Most times everything is ok and select returns 0 records. Bu= t >>>>> sometimes (~5 out of 100,000) it returns the supposedly deleted row. >>>>> >>>>> * We have logged the ExecutionInfo objects with query tracing that ar= e >>>>> returned by Datastax's driver. Here are the details >>>>> >>>>> *DELETE FROM bookingfile_uids WHERE date=3DC20131006 AND >>>>> timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e* >>>>> >>>>> *ExecutionInfo: [* >>>>> *triedHosts=3D/10.10.30.23;* >>>>> *queriedHost=3D/10.10.30.23;* >>>>> *achievedConsistencyLevel=3Dnull;* >>>>> *queryTrace=3D* >>>>> * Message received from /10.10.30.23 on /10.10.30.19[Thread-56] at >>>>> Sun Oct 06 19:55:57 MSK 2013* >>>>> * Acquiring switchLock read lock on /10.10.30.19[MutationStage:49] at >>>>> Sun Oct 06 19:55:57 MSK 2013* >>>>> * Appending to commitlog on /10.10.30.19[MutationStage:49] at Sun Oct >>>>> 06 19:55:57 MSK 2013* >>>>> * Adding to bookingfile_uids memtable on /10.10.30.19[MutationStage:4= 9] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Enqueuing response to /10.10.30.23 on /10.10.30.19[MutationStage:49= ] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Sending message to /10.10.30.23 on /10.10.30.19[WRITE-/10.10.30.23] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Message received from /10.10.30.23 on /10.10.30.20[Thread-34] at >>>>> Sun Oct 06 19:55:57 MSK 2013* >>>>> * Acquiring switchLock read lock on /10.10.30.20[MutationStage:43] at >>>>> Sun Oct 06 19:55:57 MSK 2013* >>>>> * Appending to commitlog on /10.10.30.20[MutationStage:43] at Sun Oct >>>>> 06 19:55:57 MSK 2013* >>>>> * Adding to bookingfile_uids memtable on /10.10.30.20[MutationStage:4= 3] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Enqueuing response to /10.10.30.23 on /10.10.30.20[MutationStage:43= ] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Sending message to /10.10.30.23 on /10.10.30.20[WRITE-/10.10.30.23] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Determining replicas for mutation on /10.10.30.23[Native-Transport-= Requests:1387368] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Sending message to /10.10.30.19 on /10.10.30.23[WRITE-/10.10.30.19] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Acquiring switchLock read lock on /10.10.30.23[MutationStage:46] at >>>>> Sun Oct 06 19:55:57 MSK 2013* >>>>> * Sending message to /10.10.30.20 on /10.10.30.23[WRITE-/10.10.30.20] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Message received from /10.10.30.20 on /10.10.30.23[Thread-5] at Sun >>>>> Oct 06 19:55:57 MSK 2013* >>>>> * Processing response from /10.10.30.20 on /10.10.30.23[RequestRespon= seStage:4] >>>>> at Sun Oct 06 19:55:57 MSK 2013* >>>>> * Message received from /10.10.30.19 on /10.10.30.23[Thread-7] at Sun >>>>> Oct 06 19:55:57 MSK 2013* >>>>> * Processing response from /10.10.30.19 on /10.10.30.23[RequestRespon= seStage:4] >>>>> at Sun Oct 06 19:55:57 MSK 2013;* >>>>> *]* >>>>> >>>>> *SELECT * FROM bookingfile_uids WHERE date=3DC20131006 AND >>>>> timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e returned = 1 record >>>>> * >>>>> >>>>> the same query 1 second later: >>>>> >>>>> *DELETE FROM bookingfile_uids WHERE date=3DC20131006 AND >>>>> timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e* >>>>> * >>>>> * >>>>> *ExecutionInfo: [* >>>>> *triedHosts=3D/10.10.30.20;* >>>>> *queriedHost=3D/10.10.30.20;* >>>>> *achievedConsistencyLevel=3Dnull;* >>>>> *queryTrace=3D* >>>>> * Message received from /10.10.30.20 on /10.10.30.19[Thread-57] at >>>>> Sun Oct 06 19:55:58 MSK 2013* >>>>> * Determining replicas for mutation on /10.10.30.20[Native-Transport-= Requests:1387705] >>>>> at Sun Oct 06 19:55:58 MSK 2013* >>>>> * Acquiring switchLock read lock on /10.10.30.20[MutationStage:43] at >>>>> Sun Oct 06 19:55:58 MSK 2013* >>>>> * Appending to commitlog on /10.10.30.20[MutationStage:43] at Sun Oct >>>>> 06 19:55:58 MSK 2013* >>>>> * Adding to bookingfile_uids memtable on /10.10.30.20[MutationStage:4= 3] >>>>> at Sun Oct 06 19:55:58 MSK 2013* >>>>> * Sending message to /10.10.30.19 on /10.10.30.20[WRITE-/10.10.30.19] >>>>> at Sun Oct 06 19:55:58 MSK 2013* >>>>> * Sending message to /10.10.30.23 on /10.10.30.20[WRITE-/10.10.30.23] >>>>> at Sun Oct 06 19:55:58 MSK 2013* >>>>> * Message received from /10.10.30.19 on /10.10.30.20[Thread-4] at Sun >>>>> Oct 06 19:55:58 MSK 2013* >>>>> * Processing response from /10.10.30.19 on /10.10.30.20[RequestRespon= seStage:6] >>>>> at Sun Oct 06 19:55:58 MSK 2013* >>>>> * Message received from /10.10.30.20 on /10.10.30.23[Thread-18] at >>>>> Sun Oct 06 19:55:58 MSK 2013* >>>>> *]* >>>>> * >>>>> * >>>>> *SELECT * FROM bookingfile_uids WHERE date=3DC20131006 AND >>>>> timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e returned = 0 >>>>> records.* >>>>> >>>>> * Cassandra's system.log on all 3 nodes lists nothing special during >>>>> these queries - just some compaction related *INFO* entries. >>>>> >>>>> Can anyone help with this? What is our next step? >>>>> >>>>> Thanks in advance, >>>>> Alexander >>>>> >>>> >>>> >>> >> >> > --=20 ----------------- Nate McCall Austin, TX @zznate Co-Founder & Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com --047d7b6250bc3a708104e8dd5ba3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
This is almost a guaranteed sign that the clocks are off i= n your cluster. If you run the select query a couple of times in a row righ= t after deletion, do you see the data appear again?


On Wed, Oct 16, 2013 at 12:12 AM, Alexan= der Shutyaev <shutyaev@gmail.com> wrote:
Hi all,

Unfortunately, we still have a = problem. I've modified my code, so that it explicitly sets the consiste= ncy level to QUORUM for each query. However, we found out a few case= s when the record is deleted on only 1 node of 3. In this cases the = delete query executed ok, and the select query that we do rig= ht after delete returned 0 rows. Later when we ran a global daily ch= eck select returned 1 row. How can that be? What can we be mi= ssing?

2013/10/7 Jon Haddad <= ;jon@jonhaddad.com>
I haven't used VMWare but it s= eems odd that it would lock up the ntp port. =C2=A0try "ps aux | grep = ntp" to see if ntpd it's already running.


Hi Micha=C5=82,

I didn't notic= e your message at first.. Well this seems like a real cause candidate.. I&#= 39;ll add an explicit consistency level QUORUM and see if that helps. Thank= s


2013/10= /7 Alexander Shutyaev <shutyaev@gmail.com>
Hi Nick,

Thanks for the note! We = have our cassanra instances installed on virtual hosts in VMWare and the cl= ock synchronization is handled by the latter, so I can't use ntpdate (s= ays that NTP socket is in use). Is there any way to check if the clocks are= really synchronized? My best attempt was using three shell windows with co= mmands already typed thus requiring only clicking on the window and hitting= enter. The results varied by 100-200 msec which I guess is just about the = time I need to click and press enter :)

Thanks in advance,
Alexander
=


2013/10/7 Nik= olay Mihaylov <nmmm@nmmm.nu>
Hi

my tw= o cents - before doing anything else, make sure clocks are synchronized to = the millisecond.
ntp will do so.

Nick.


On Mon, Oct 7, 2013 at 9:02 AM, Alexande= r Shutyaev <shutyaev@gmail.com> wrote:
Hi all,

We have encountered the followi= ng problem with cassandra.

* We use cassandra v= 2.0.0 from Datastax community repo.

* We have 3 nodes in a cluster, all of them are seed = providers.

* We have a single keyspace with= replication factor =3D 3:

CREATE KEYSPACE bof WITH replication =3D {
= =C2=A0 'class': 'SimpleStrategy',
=C2=A0 &= #39;replication_factor': '3'
};
=
* We use Datastax Java CQL Driver v1.0.3 in our applicati= on.

* We have not modified any consistency sett= ings in our app, so I assume we have the default QUORUM (2 out o= f 3 in our case) consistency for reads and writes.

* We have 400+ tables which can be divided in two group= s (main and uids). All tables in a group have the same defini= tion, they vary only by name. The sample definitions are:

CREATE TABLE bookingfile (
=C2= =A0 key text,
=C2=A0 entity_created timestamp,
=C2=A0 entity_createdby text,
=C2=A0 entity_entityty= pe text,
=C2=A0 entity_modified timestamp,
=C2=A0 entity_mo= difiedby text,
=C2=A0 entity_status text,
=C2=A0 entity_uid text,
=C2=A0 entity_updatepolicy text,<= /b>
=C2=A0 version_created timestamp,
=C2=A0 version_createdby text,
=C2=A0 version_data= blob,
=C2=A0 version_dataformat text,
= =C2=A0 version_datasource text,
=C2=A0 version_modified ti= mestamp,
=C2=A0 version_modifiedby text,
=C2=A0 version_uid text= ,
=C2=A0 version_versionnotes text,
=C2= =A0 version_versionnumber int,
=C2=A0 versionscount int,
=C2=A0 PRIMARY KEY (key)
) WITH
=C2=A0 bloom_filter_fp_chance=3D0.010000 AN= D
=C2=A0 caching=3D'KEYS_ONLY' AND
<= b>=C2=A0 comment=3D'' AND
=C2=A0 dclocal_read_repa= ir_chance=3D0.000000 AND
=C2=A0 gc_grace_seconds=3D864000 AND
=C2=A0 index_= interval=3D128 AND
=C2=A0 read_repair_chance=3D0.100000 AN= D
=C2=A0 replicate_on_write=3D'true' AND
=
=C2=A0 populate_io_cache_on_flush=3D'false' AND
=C2=A0 default_time_to_live=3D0 AND
=C2=A0 specula= tive_retry=3D'NONE' AND
=C2=A0 memtable_flush_peri= od_in_ms=3D0 AND
=C2=A0 compaction=3D{'class': = 9;SizeTieredCompactionStrategy'} AND
=C2=A0 compression=3D{'sstable_compression': 'LZ4Compre= ssor'};

CREATE TABLE bookingfile_u= ids (
=C2=A0 date text,
=C2=A0 timeanduid= text,
=C2=A0 deleted boolean,
=C2=A0 PRIMARY KEY (date, = timeanduid)
) WITH
=C2=A0 bloom_filter_fp= _chance=3D0.010000 AND
=C2=A0 caching=3D'KEYS_ONLY'= ; AND
=C2=A0 comment=3D'' AND
=C2=A0 dclocal_read_rep= air_chance=3D0.000000 AND
=C2=A0 gc_grace_seconds=3D864000= AND
=C2=A0 index_interval=3D128 AND
=C2= =A0 read_repair_chance=3D0.100000 AND
=C2=A0 replicate_on_write=3D'true' AND
=C2= =A0 populate_io_cache_on_flush=3D'false' AND
=C2= =A0 default_time_to_live=3D0 AND
=C2=A0 speculative_retry= =3D'NONE' AND
=C2=A0 memtable_flush_period_in_ms=3D0 AND
=C2=A0 = compaction=3D{'class': 'SizeTieredCompactionStrategy'} AND<= /b>
=C2=A0 compression=3D{'sstable_compression': '= LZ4Compressor'};

CREATE INDEX BookingFile_uids_deleted ON book= ingfile_uids (deleted);

* We don't h= ave any problems with the tables from the main group.

* As for the tables from the uids=C2=A0group we = have noticed that sometimes deletes from these tables do not do their job. = They don't fail, they just do nothing. We have confirmed this by adding= a select query after deletes. Most times everything is ok and select retur= ns 0 records. But sometimes (~5 out of 100,000) it returns the supposedly d= eleted row.

* We have logged the ExecutionInfo objects with query t= racing that are returned by Datastax's driver. Here are the details

DELETE FROM bookingfile_uids WHERE date=3DC2013100= 6 AND timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e

ExecutionInfo: [
triedHosts=3D= /10.10.30.23;
queriedHost=3D/= 10.10.30.23;
achievedConsistencyLevel=3Dnull;
queryTrace=3D
Message received from= /10.10.30.23 on /10.10.30.19[Thread-56] a= t Sun Oct 06 19:55:57 MSK 2013
Appending to commitlog on /10.10.30.19[MutationStage:49= ] at Sun Oct 06 19:55:57 MSK 2013
Adding to bookingfile_= uids memtable on /10.10.3= 0.19[MutationStage:49] at Sun Oct 06 19:55:57 MSK 2013
Enqueuing response to /10.10.30.23 on /10.10.30.19[MutationStage:49] at= Sun Oct 06 19:55:57 MSK 2013
Sending message to /10.10.30.23 on /10.10.30.19[WRITE-/10.10.30.23] at Sun Oct 06 1= 9:55:57 MSK 2013
Message received from = /10.10.30.23 on /10.10.30.20[Thread-34] at= Sun Oct 06 19:55:57 MSK 2013
Appending to commitlog on /10.10.30.20[MutationStage:43= ] at Sun Oct 06 19:55:57 MSK 2013
Adding to bookingfile_= uids memtable on /10.10.3= 0.20[MutationStage:43] at Sun Oct 06 19:55:57 MSK 2013
Enqueuing response to /10.10.30.23 on /10.10.30.20[MutationStage:43] at= Sun Oct 06 19:55:57 MSK 2013
Sending message to /10.10.30.23 on /10.10.30.20[WRITE-/10.10.30.23] at Sun Oct 06 1= 9:55:57 MSK 2013
Determining replicas f= or mutation on /10.10.30.= 23[Native-Transport-Requests:1387368] at Sun Oct 06 19:55:57 MSK 2013
Sending message to /10.10.30.19 on /10.10.30.23[WRITE-/10.10.30.19] at Sun Oct 06 1= 9:55:57 MSK 2013
Sending message to /10.10.30.20 on /10.10.30.23[WRITE-/10.10.30.20] at Sun Oct 06 19:55:57 MSK 20= 13
Message received from = /10.10.30.20 on /10.10.30.23[Thread-5] at = Sun Oct 06 19:55:57 MSK 2013
Processing response fr= om /10.10.30.20 on /<= a href=3D"http://10.10.30.23/" target=3D"_blank">10.10.30.23[RequestRes= ponseStage:4] at Sun Oct 06 19:55:57 MSK 2013
Message received from = /10.10.30.19 on /10.10.30.23[Thread-7] at = Sun Oct 06 19:55:57 MSK 2013
Processing response fr= om /10.10.30.19 on /<= a href=3D"http://10.10.30.23/" target=3D"_blank">10.10.30.23[RequestRes= ponseStage:4] at Sun Oct 06 19:55:57 MSK 2013;
]

SELECT * FROM bookingfile_uids WH= ERE date=3DC20131006 AND timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7f= cb6b7394e returned 1 record

the same query= 1 second later:

DELETE FROM bookingfile_uids WHERE date=3DC2013= 1006 AND timeAndUid=3D195248590_4762ce41-d2d2-448d-be8c-c7fcb6b7394e

ExecutionInfo: [
triedHosts=3D/10.10.30= .20;
queriedHost=3D/10.10.30.20;
achievedConsistencyLevel= =3Dnull;
queryTrace=3D
Message received from = /10.10.30.20 on /10.10.30.19[Thread-57] at= Sun Oct 06 19:55:58 MSK 2013
Determining replicas f= or mutation on /10.10.30.= 20[Native-Transport-Requests:1387705] at Sun Oct 06 19:55:58 MSK 2013
Appending to commitlog= on /10.10.30.20[Muta= tionStage:43] at Sun Oct 06 19:55:58 MSK 2013
Adding to bookingfile_uids memtable on /<= a href=3D"http://10.10.30.20/" target=3D"_blank">10.10.30.20[MutationSt= age:43] at Sun Oct 06 19:55:58 MSK 2013
Sending message to /10.10.30.19 on /10.10.30.20[WRITE-/10.10.30.19] at Sun Oct 06 1= 9:55:58 MSK 2013
Sending message to /10.10.30.23 on /10.10.30.20[WRITE-/10.10.30.23] at Sun Oct 06 1= 9:55:58 MSK 2013
Message received from = /10.10.30.19 on /10.10.30.20[Thread-4] at = Sun Oct 06 19:55:58 MSK 2013
Processing response fr= om /10.10.30.19 on /<= a href=3D"http://10.10.30.20/" target=3D"_blank">10.10.30.20[RequestRes= ponseStage:6] at Sun Oct 06 19:55:58 MSK 2013
Message received from = /10.10.30.20 on /10.10.30.23[Thread-18] at= Sun Oct 06 19:55:58 MSK 2013
]

SELECT * FROM bookingfile_= uids WHERE date=3DC20131006 AND timeAndUid=3D195248590_4762ce41-d2d2-448d-b= e8c-c7fcb6b7394e returned 0 records.

* Cassand= ra's system.log on all 3 nodes lists nothing special during these queri= es - just some compaction related INFO entries.

Can anyone help with this? What is our next step?
=

Thanks in advance,
Alexander








--
=
-----------------
Nate McCall
Austin, TX
@zznate<= br>
Co-Founder & Sr. Technical Consultant
Apache Cassandra Consul= ting
http://www.thela= stpickle.com
--047d7b6250bc3a708104e8dd5ba3--