Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates
 209.85.218.222 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=nFa4mVjSfem+3xD9iwvaEK5tFYShCtWFoChYdQdO+ViKDGA/td9ph5IdhS0kTLXA/F
         03ZtvjbomYqnUXpHa8WTYluRW7vx9QhzuBKPaLW5VVs/YJYRnqkrxept24iS9hSIUqer
         M/NCtQtF3esW2zPmulR19sqB7dSNUP0S9wObc=
MIME-Version: 1.0
In-Reply-To: <h2g775e31411004281356v36f73707nab94e3dbd03fa725@mail.gmail.com>
References: <m2x775e31411004241023h7a79739o2834870c16976190@mail.gmail.com>
	<r2se06563881004261845v2e1c6dcdrd2b8aa407794733e@mail.gmail.com>
	<i2z775e31411004271044wf387a006ha564ac18d6245a08@mail.gmail.com>
	<h2p775e31411004271420zabde269ct7905419a45724ea@mail.gmail.com>
	<r2ia17e13e71004271435w2ba90b14xeb52179f7173b3f3@mail.gmail.com>
	<w2w775e31411004271437hb07da656o12d477084d188663@mail.gmail.com>
	<l2j775e31411004271514te2b3c531qc31d19eb34112f92@mail.gmail.com>
	<w2qe06563881004280743ocbefed2dk2031dc2de3796276@mail.gmail.com>
	<h2g775e31411004281356v36f73707nab94e3dbd03fa725@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Wed, 28 Apr 2010 23:51:49 -0500
Message-ID: <g2xe06563881004282151zbc1cf5fbldb0e70efd30ccde0@mail.gmail.com>
Subject: Re: Cassandra reverting deletes?
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Good! :)

Can you reproduce w/o map/reduce, with raw get_range_slices?

On Wed, Apr 28, 2010 at 3:56 PM, Joost Ouwerkerk <joost@openplaces.org> wro=
te:
> Yes! Reproduced on single-node cluster:
>
> 10/04/28 16:30:24 INFO mapred.JobClient: =A0 =A0 ROWS=3D274884
> 10/04/28 16:30:24 INFO mapred.JobClient: =A0 =A0 TOMBSTONES=3D951083
>
> 10/04/28 16:42:49 INFO mapred.JobClient: =A0 =A0 ROWS=3D166580
> 10/04/28 16:42:49 INFO mapred.JobClient: =A0 =A0 TOMBSTONES=3D1059387
>
> On Wed, Apr 28, 2010 at 10:43 AM, Jonathan Ellis <jbellis@gmail.com> wrot=
e:
>> It sounds like either there is a fairly obvious bug, or you're doing
>> something wrong. :)
>>
>> Can you reproduce against a single node?
>>
>> On Tue, Apr 27, 2010 at 5:14 PM, Joost Ouwerkerk <joost@openplaces.org> =
wrote:
>>> Update: I ran a test whereby I deleted ALL the rows in a column
>>> family, using a consistency level of ALL. =A0To do this, I mapped the
>>> ColumnFamily and called remove on each row id. =A0There were 1.5 millio=
n
>>> rows, so 1.5 million rows were deleted.
>>>
>>> I ran a counter job immediately after. =A0This job maps the same column
>>> family and tests if any data is returned. =A0If not, it considers the
>>> row a "tombstone". =A0If yes, it considers the row not deleted. =A0Belo=
w
>>> are the hadoop counters for those jobs. =A0Note the fluctuation in the
>>> number of rows with data over time, and the increase in time to map
>>> the column family after the destroy job. =A0No other clients were
>>> accessing cassandra during this time.
>>>
>>> I'm thoroughly confused.
>>>
>>> Count: started 13:02:30 EDT, finished 13:11:33 EDT (9 minutes 2 seconds=
):
>>> =A0 ROWS: =A0 =A0 =A0 =A01,542,479
>>> =A0 TOMBSTONES: =A069
>>>
>>> Destroy: started 16:48:45 EDT, finished 17:07:36 EDT (18 minutes 50 sec=
onds)
>>> =A0 DESTROYED: =A01,542,548
>>>
>>> Count: started 17:15:42 EDT, finished 17:31:03 EDT (15 minutes 21 secon=
ds)
>>> =A0 ROWS 876,464
>>> =A0 TOMBSTONES =A0 666,084
>>>
>>> Count: started 17:31:32, finished 17:47:16 (15mins, 44 seconds)
>>> =A0 ROWS 1,451,665
>>> =A0 TOMBSTONES =A0 90,883
>>>
>>> Count: started 17:52:34, finished 18:10:28 (17mins, 53 seconds)
>>> =A0 ROWS 1,425,644
>>> =A0 TOMBSTONES =A0 116,904
>>>
>>> On Tue, Apr 27, 2010 at 5:37 PM, Joost Ouwerkerk <joost@openplaces.org>=
 wrote:
>>>> Clocks are in sync:
>>>>
>>>> cluster04:~/cassandra$ dsh -g development "date"
>>>> Tue Apr 27 17:36:33 EDT 2010
>>>> Tue Apr 27 17:36:33 EDT 2010
>>>> Tue Apr 27 17:36:33 EDT 2010
>>>> Tue Apr 27 17:36:33 EDT 2010
>>>> Tue Apr 27 17:36:34 EDT 2010
>>>> Tue Apr 27 17:36:34 EDT 2010
>>>> Tue Apr 27 17:36:34 EDT 2010
>>>> Tue Apr 27 17:36:34 EDT 2010
>>>> Tue Apr 27 17:36:34 EDT 2010
>>>> Tue Apr 27 17:36:35 EDT 2010
>>>> Tue Apr 27 17:36:35 EDT 2010
>>>> Tue Apr 27 17:36:35 EDT 2010
>>>>
>>>> On Tue, Apr 27, 2010 at 5:35 PM, Nathan McCall <nate@vervewireless.com=
> wrote:
>>>>> Have you confirmed that your clocks are all synced in the cluster?
>>>>> This may be the result of an unintentional read-repair occurring if
>>>>> that were the case.
>>>>>
>>>>> -Nate
>>>>>
>>>>> On Tue, Apr 27, 2010 at 2:20 PM, Joost Ouwerkerk <joost@openplaces.or=
g> wrote:
>>>>>> Hmm... Even after deleting with cl.ALL, I'm getting data back for so=
me
>>>>>> rows after having deleted them. =A0Which rows return data is
>>>>>> inconsistent from one run of the job to the next.
>>>>>>
>>>>>> On Tue, Apr 27, 2010 at 1:44 PM, Joost Ouwerkerk <joost@openplaces.o=
rg> wrote:
>>>>>>> To check that rows are gone, I check that KeySlice.columns is empty=
. =A0And as
>>>>>>> I mentioned, immediately after the delete job, this returns the exp=
ected
>>>>>>> number.
>>>>>>> Unfortunately I reproduced with QUORUM this morning. =A0No node out=
ages. =A0I am
>>>>>>> going to try ALL to see if that changes anything, but I am starting=
 to
>>>>>>> wonder if I'm doing something else wrong.
>>>>>>> On Mon, Apr 26, 2010 at 9:45 PM, Jonathan Ellis <jbellis@gmail.com>=
 wrote:
>>>>>>>>
>>>>>>>> How are you checking that the rows are gone?
>>>>>>>>
>>>>>>>> Are you experiencing node outages during this?
>>>>>>>>
>>>>>>>> DC_QUORUM is unfinished code right now, you should avoid using it.
>>>>>>>> Can you reproduce with normal QUORUM?
>>>>>>>>
>>>>>>>> On Sat, Apr 24, 2010 at 12:23 PM, Joost Ouwerkerk <joost@openplace=
s.org>
>>>>>>>> wrote:
>>>>>>>> > I'm having trouble deleting rows in Cassandra.=A0 After running =
a job that
>>>>>>>> > deletes hundreds of rows, I run another job that verifies that t=
he rows
>>>>>>>> > are
>>>>>>>> > gone.=A0 Both jobs run correctly.=A0 However, when I run the ver=
ification
>>>>>>>> > job an
>>>>>>>> > hour later, the rows have re-appeared.=A0 This is not a case of =
"ghosting"
>>>>>>>> > because the verification job actually checks that there is data =
in the
>>>>>>>> > columns.
>>>>>>>> >
>>>>>>>> > I am running a cluster with 12 nodes and a replication factor of=
 3.=A0 I
>>>>>>>> > am
>>>>>>>> > using DC_QUORUM consistency when deleting.
>>>>>>>> >
>>>>>>>> > Any ideas?
>>>>>>>> > Joost.
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jonathan Ellis
>>>>>>>> Project Chair, Apache Cassandra
>>>>>>>> co-founder of Riptano, the source for professional Cassandra suppo=
rt
>>>>>>>> http://riptano.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com