Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 31072 invoked from network); 29 Apr 2010 23:55:54 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 29 Apr 2010 23:55:54 -0000 Received: (qmail 11820 invoked by uid 500); 29 Apr 2010 23:55:53 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 11799 invoked by uid 500); 29 Apr 2010 23:55:53 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 11791 invoked by uid 99); 29 Apr 2010 23:55:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 23:55:53 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=AWL,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.160.172] (HELO mail-gy0-f172.google.com) (209.85.160.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 23:55:49 +0000 Received: by gyh4 with SMTP id 4so8478766gyh.31 for ; Thu, 29 Apr 2010 16:55:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.90.2.18 with SMTP id 18mr97804agb.7.1272585326311; Thu, 29 Apr 2010 16:55:26 -0700 (PDT) Received: by 10.90.55.19 with HTTP; Thu, 29 Apr 2010 16:55:26 -0700 (PDT) In-Reply-To: References: Date: Thu, 29 Apr 2010 19:55:26 -0400 Message-ID: Subject: Re: Cassandra reverting deletes? From: Joost Ouwerkerk To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Ok, I reproduced without mapred. Here is my recipe: On a single-node cassandra cluster with basic config (-Xmx:1G) loop { * insert 5,000 records in a single columnfamily with UUID keys and random string values (between 1 and 1000 chars) in 5 different columns spanning two different supercolumns * delete all the data by iterating over the rows with get_range_slices(ONE) and calling remove(QUORUM) on each row id returned (path containing only columnfamily) * count number of non-tombstone rows by iterating over the rows with get_range_slices(ONE) and testing data. Break if not zero. } Here's the flakey part: while this is running, call "bin/nodetool -h localhost -p 8081 flush KeySpace" in the background every minute or so. When the data hits some critical size, the loop will break. Anyone care to try this at home? On Thu, Apr 29, 2010 at 12:51 AM, Jonathan Ellis wrote: > Good! :) > > Can you reproduce w/o map/reduce, with raw get_range_slices? > > On Wed, Apr 28, 2010 at 3:56 PM, Joost Ouwerkerk w= rote: >> Yes! Reproduced on single-node cluster: >> >> 10/04/28 16:30:24 INFO mapred.JobClient: =A0 =A0 ROWS=3D274884 >> 10/04/28 16:30:24 INFO mapred.JobClient: =A0 =A0 TOMBSTONES=3D951083 >> >> 10/04/28 16:42:49 INFO mapred.JobClient: =A0 =A0 ROWS=3D166580 >> 10/04/28 16:42:49 INFO mapred.JobClient: =A0 =A0 TOMBSTONES=3D1059387 >> >> On Wed, Apr 28, 2010 at 10:43 AM, Jonathan Ellis wro= te: >>> It sounds like either there is a fairly obvious bug, or you're doing >>> something wrong. :) >>> >>> Can you reproduce against a single node? >>> >>> On Tue, Apr 27, 2010 at 5:14 PM, Joost Ouwerkerk = wrote: >>>> Update: I ran a test whereby I deleted ALL the rows in a column >>>> family, using a consistency level of ALL. =A0To do this, I mapped the >>>> ColumnFamily and called remove on each row id. =A0There were 1.5 milli= on >>>> rows, so 1.5 million rows were deleted. >>>> >>>> I ran a counter job immediately after. =A0This job maps the same colum= n >>>> family and tests if any data is returned. =A0If not, it considers the >>>> row a "tombstone". =A0If yes, it considers the row not deleted. =A0Bel= ow >>>> are the hadoop counters for those jobs. =A0Note the fluctuation in the >>>> number of rows with data over time, and the increase in time to map >>>> the column family after the destroy job. =A0No other clients were >>>> accessing cassandra during this time. >>>> >>>> I'm thoroughly confused. >>>> >>>> Count: started 13:02:30 EDT, finished 13:11:33 EDT (9 minutes 2 second= s): >>>> =A0 ROWS: =A0 =A0 =A0 =A01,542,479 >>>> =A0 TOMBSTONES: =A069 >>>> >>>> Destroy: started 16:48:45 EDT, finished 17:07:36 EDT (18 minutes 50 se= conds) >>>> =A0 DESTROYED: =A01,542,548 >>>> >>>> Count: started 17:15:42 EDT, finished 17:31:03 EDT (15 minutes 21 seco= nds) >>>> =A0 ROWS 876,464 >>>> =A0 TOMBSTONES =A0 666,084 >>>> >>>> Count: started 17:31:32, finished 17:47:16 (15mins, 44 seconds) >>>> =A0 ROWS 1,451,665 >>>> =A0 TOMBSTONES =A0 90,883 >>>> >>>> Count: started 17:52:34, finished 18:10:28 (17mins, 53 seconds) >>>> =A0 ROWS 1,425,644 >>>> =A0 TOMBSTONES =A0 116,904 >>>> >>>> On Tue, Apr 27, 2010 at 5:37 PM, Joost Ouwerkerk wrote: >>>>> Clocks are in sync: >>>>> >>>>> cluster04:~/cassandra$ dsh -g development "date" >>>>> Tue Apr 27 17:36:33 EDT 2010 >>>>> Tue Apr 27 17:36:33 EDT 2010 >>>>> Tue Apr 27 17:36:33 EDT 2010 >>>>> Tue Apr 27 17:36:33 EDT 2010 >>>>> Tue Apr 27 17:36:34 EDT 2010 >>>>> Tue Apr 27 17:36:34 EDT 2010 >>>>> Tue Apr 27 17:36:34 EDT 2010 >>>>> Tue Apr 27 17:36:34 EDT 2010 >>>>> Tue Apr 27 17:36:34 EDT 2010 >>>>> Tue Apr 27 17:36:35 EDT 2010 >>>>> Tue Apr 27 17:36:35 EDT 2010 >>>>> Tue Apr 27 17:36:35 EDT 2010 >>>>> >>>>> On Tue, Apr 27, 2010 at 5:35 PM, Nathan McCall wrote: >>>>>> Have you confirmed that your clocks are all synced in the cluster? >>>>>> This may be the result of an unintentional read-repair occurring if >>>>>> that were the case. >>>>>> >>>>>> -Nate >>>>>> >>>>>> On Tue, Apr 27, 2010 at 2:20 PM, Joost Ouwerkerk wrote: >>>>>>> Hmm... Even after deleting with cl.ALL, I'm getting data back for s= ome >>>>>>> rows after having deleted them. =A0Which rows return data is >>>>>>> inconsistent from one run of the job to the next. >>>>>>> >>>>>>> On Tue, Apr 27, 2010 at 1:44 PM, Joost Ouwerkerk wrote: >>>>>>>> To check that rows are gone, I check that KeySlice.columns is empt= y. =A0And as >>>>>>>> I mentioned, immediately after the delete job, this returns the ex= pected >>>>>>>> number. >>>>>>>> Unfortunately I reproduced with QUORUM this morning. =A0No node ou= tages. =A0I am >>>>>>>> going to try ALL to see if that changes anything, but I am startin= g to >>>>>>>> wonder if I'm doing something else wrong. >>>>>>>> On Mon, Apr 26, 2010 at 9:45 PM, Jonathan Ellis wrote: >>>>>>>>> >>>>>>>>> How are you checking that the rows are gone? >>>>>>>>> >>>>>>>>> Are you experiencing node outages during this? >>>>>>>>> >>>>>>>>> DC_QUORUM is unfinished code right now, you should avoid using it= . >>>>>>>>> Can you reproduce with normal QUORUM? >>>>>>>>> >>>>>>>>> On Sat, Apr 24, 2010 at 12:23 PM, Joost Ouwerkerk >>>>>>>>> wrote: >>>>>>>>> > I'm having trouble deleting rows in Cassandra.=A0 After running= a job that >>>>>>>>> > deletes hundreds of rows, I run another job that verifies that = the rows >>>>>>>>> > are >>>>>>>>> > gone.=A0 Both jobs run correctly.=A0 However, when I run the ve= rification >>>>>>>>> > job an >>>>>>>>> > hour later, the rows have re-appeared.=A0 This is not a case of= "ghosting" >>>>>>>>> > because the verification job actually checks that there is data= in the >>>>>>>>> > columns. >>>>>>>>> > >>>>>>>>> > I am running a cluster with 12 nodes and a replication factor o= f 3.=A0 I >>>>>>>>> > am >>>>>>>>> > using DC_QUORUM consistency when deleting. >>>>>>>>> > >>>>>>>>> > Any ideas? >>>>>>>>> > Joost. >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Jonathan Ellis >>>>>>>>> Project Chair, Apache Cassandra >>>>>>>>> co-founder of Riptano, the source for professional Cassandra supp= ort >>>>>>>>> http://riptano.com >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >>> >>> -- >>> Jonathan Ellis >>> Project Chair, Apache Cassandra >>> co-founder of Riptano, the source for professional Cassandra support >>> http://riptano.com >>> >> > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >