Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 17AF691BF for ; Wed, 7 Mar 2012 09:42:22 +0000 (UTC) Received: (qmail 55062 invoked by uid 500); 7 Mar 2012 09:42:19 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 55052 invoked by uid 500); 7 Mar 2012 09:42:19 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 55043 invoked by uid 99); 7 Mar 2012 09:42:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Mar 2012 09:42:18 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [94.75.197.170] (HELO mail.unitedgames.com) (94.75.197.170) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Mar 2012 09:42:13 +0000 Received: from localhost (localhost [127.0.0.1]) by mail.unitedgames.com (Postfix) with ESMTP id 2FEE32D86A for ; Wed, 7 Mar 2012 10:41:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=unitedgames.com; h=content-type:content-type:in-reply-to:references:subject :subject:to:mime-version:user-agent:from:from:date:date :message-id; s=dkim; t=1331113301; x=1331977301; bh=8jGrrgI6/lS0 b60aI3DF2s6soISQ0EAbZDxDPanitYc=; b=gNhun6sPy4oAXeQ9vlUGt4fwzWIG lWE5gtisdf6yzQTef7chB9dLp4003zZbkZzJI1JK5pD5dHk1x8T0z/wBRIRqTlTd B5hFrDfcmf6uLK6bwl7j8P/JrKMRDdmEhlQfDcVnwrMoJOsTTpChTII4HjPsX5CE zJ1VT2zntUmc0jQ= X-Virus-Scanned: Debian amavisd-new at mail.unitedgames.com Received: from mail.unitedgames.com ([127.0.0.1]) by localhost (mail.unitedgames.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9qblaRggLiBS for ; Wed, 7 Mar 2012 10:41:41 +0100 (CET) Received: from [192.168.1.92] (unknown [188.204.191.147]) by mail.unitedgames.com (Postfix) with ESMTPA id 05CB02D865 for ; Wed, 7 Mar 2012 10:41:40 +0100 (CET) Message-ID: <4F572D50.3050500@unitedgames.com> Date: Wed, 07 Mar 2012 10:41:36 +0100 From: Stefan Reek User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.16) Gecko/20110307 Iceowl/1.0b1 Icedove/3.0.11 MIME-Version: 1.0 To: user@cassandra.apache.org Subject: Re: Old data coming alive after adding node References: <4F55D522.7080807@unitedgames.com> <745045D3-5C52-4B71-BFF4-2B14EEBC6EFA@thelastpickle.com> <4F55E362.3080307@unitedgames.com> <0679AE72-BB02-4B19-ADE3-6DE95932E35F@thelastpickle.com> In-Reply-To: <0679AE72-BB02-4B19-ADE3-6DE95932E35F@thelastpickle.com> Content-Type: multipart/alternative; boundary="------------030505020104040105020301" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------030505020104040105020301 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit After the old data came up we were able to delete it again. And it is stable now. We are in the process of upgrading to 1.0, but as you said that's a painful process. I just hope 0.6 will keep running till we're done with the upgrade. Anyway thanks for the help. Cheers, Stefan On 03/06/2012 07:02 PM, aaron morton wrote: >> All our writes/deletes are done with CL.QUORUM. >> Our reads are done with CL.ONE. Although the reads that confirmed the >> old data were done with CL.QUORUM. > mmmm > >> According to >> https://svn.apache.org/viewvc/cassandra/branches/cassandra-0.6/CHANGES.txt 0.6.6 >> has the same patch >> for (CASSANDRA-1074) as 0.7 and so I assumed that minor compactions >> in 0.6.6 and up also purged tombstones. > My bad. As you were. > > After the repair did the un-deleted data remain un-deleted ? Are you > back to a stable situation ? > > Without a lot more detail I am at a bit of a loss. > > I know it's painful but migrating to 1.0 *really* will make your life > so much easier and faster. At some point you may hit a bug or a > problem in 0.6 and the solution may be to upgrade, quickly. > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 6/03/2012, at 11:13 PM, Stefan Reek wrote: > >> Hi Aaron, >> >> Thanks for the quick reply. >> All our writes/deletes are done with CL.QUORUM. >> Our reads are done with CL.ONE. Although the reads that confirmed the >> old data were done with CL.QUORUM. >> According to >> https://svn.apache.org/viewvc/cassandra/branches/cassandra-0.6/CHANGES.txt >> 0.6.6 has the same patch >> for (CASSANDRA-1074) as 0.7 and so I assumed that minor compactions >> in 0.6.6 and up also purged tombstones. >> The only suspicious thing I noticed was that after adding the fourth >> node repairs became extremely slow and heavy. >> Running it degraded the performance of the whole cluster and the new >> node even went OOM when running it. >> >> Cheers, >> >> Stefan >> >> On 03/06/2012 10:51 AM, aaron morton wrote: >>>> After we added a fourth node, keeping RF=3, some old data appeared >>>> in the database. >>> What CL are you working at ? (Should not matter too much with repair >>> working, just asking) >>> >>> >>>> We don't run compact on the nodes explicitly as I understand that >>>> running repair will trigger a >>>> major compaction. I'm not entirely sure if it does so, but in any >>>> case the tombstones will be removed by a minor >>>> compaction. >>> In 0.6.x tombstones were only purged during a major / manual >>> compaction. Purging during minor compaction came in during 0.7 >>> https://github.com/apache/cassandra/blob/trunk/CHANGES.txt#L1467 >>> >>>> Can anyone think of any reason why the old data reappeared? >>> It sounds like you are doing things correctly. The complicating >>> factor is 0.6 is so very old. >>> >>> >>> If I wanted to poke around some more I would conduct reads as CL one >>> against nodes and see if they return the "deleted" data or not. This >>> would help me understand if the tombstone is still out there. >>> >>> I would also poke around a lot in the logs to make sure repair was >>> running as expected and completing. If you find anything suspicious >>> post examples. >>> >>> Finally I would ensure CL QUROUM was been used. >>> >>> Hope that helps. >>> >>> >>> ----------------- >>> Aaron Morton >>> Freelance Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 6/03/2012, at 10:13 PM, Stefan Reek wrote: >>> >>>> Hi, >>>> >>>> We were running a 3-node cluster of cassandra 0.6.13 with RF=3. >>>> After we added a fourth node, keeping RF=3, some old data appeared >>>> in the database. >>>> As far as I understand this can only happen if nodetool repair >>>> wasn't run for more than GCGraceSeconds. >>>> Our GCGraceSeconds is set to the default of 10 days (864000 seconds). >>>> We have a scheduled cronjob to run repair once each week on every >>>> node, each on another day. >>>> I'm sure that none of the nodes ever skipped running a repair. >>>> We don't run compact on the nodes explicitly as I understand that >>>> running repair will trigger a >>>> major compaction. I'm not entirely sure if it does so, but in any >>>> case the tombstones will be removed by a minor >>>> compaction. So I expected that the reappearing data, which is a >>>> couple of months old in some cases, was long gone >>>> by the time we added the node. >>>> >>>> Can anyone think of any reason why the old data reappeared? >>>> >>>> Stefan >>> >> > --------------030505020104040105020301 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit After the old data came up we were able to delete it again. And it is stable now.
We are in the process of upgrading to 1.0, but as you said that's a painful process.
I just hope 0.6 will keep running till we're done with the upgrade.
Anyway thanks for the help.

Cheers,

Stefan



On 03/06/2012 07:02 PM, aaron morton wrote:
All our writes/deletes are done with CL.QUORUM.
Our reads are done with CL.ONE. Although the reads that confirmed the old data were done with CL.QUORUM.
mmmm

According to https://svn.apache.org/viewvc/cassandra/branches/cassandra-0.6/CHANGES.txt 0.6.6 has the same patch
for (CASSANDRA-1074) as 0.7 and so I assumed that minor compactions in 0.6.6 and up also purged tombstones.
My bad. As you were. 

After the repair did the un-deleted data remain un-deleted ? Are you back to a stable situation ? 

Without a lot more detail I am at a bit of a loss. 

I know it's painful but migrating to 1.0 *really* will make your life so much easier and faster. At some point you may hit a bug or a problem in 0.6 and the solution may be to upgrade, quickly.

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 6/03/2012, at 11:13 PM, Stefan Reek wrote:

Hi Aaron,

Thanks for the quick reply.
All our writes/deletes are done with CL.QUORUM.
Our reads are done with CL.ONE. Although the reads that confirmed the old data were done with CL.QUORUM.
According to https://svn.apache.org/viewvc/cassandra/branches/cassandra-0.6/CHANGES.txt 0.6.6 has the same patch
for (CASSANDRA-1074) as 0.7 and so I assumed that minor compactions in 0.6.6 and up also purged tombstones.
The only suspicious thing I noticed was that after adding the fourth node repairs became extremely slow and heavy.
Running it degraded the performance of the whole cluster and the new node even went OOM when running it.

Cheers,

Stefan

On 03/06/2012 10:51 AM, aaron morton wrote:
After we added a fourth node, keeping RF=3, some old data appeared in the database.
What CL are you working at ? (Should not matter too much with repair working, just asking)


We don't run compact on the nodes explicitly as I understand that running repair will trigger a
major compaction. I'm not entirely sure if it does so, but in any case the tombstones will be removed by a minor
compaction.
In 0.6.x tombstones were only purged during a major / manual compaction. Purging during minor compaction came in during 0.7
https://github.com/apache/cassandra/blob/trunk/CHANGES.txt#L1467

Can anyone think of any reason why the old data reappeared?
It sounds like you are doing things correctly. The complicating factor is 0.6 is so very old. 


If I wanted to poke around some more I would conduct reads as CL one against nodes and see if they return the "deleted" data or not. This would help me understand if the tombstone is still out there. 

I would also poke around a lot in the logs to make sure repair was running as expected and completing. If you find anything suspicious post examples. 

Finally I would ensure CL QUROUM was been used. 

Hope that helps.


-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 6/03/2012, at 10:13 PM, Stefan Reek wrote:

Hi,

We were running a 3-node cluster of cassandra 0.6.13 with RF=3.
After we added a fourth node, keeping RF=3, some old data appeared in the database.
As far as I understand this can only happen if nodetool repair wasn't run for more than GCGraceSeconds.
Our GCGraceSeconds is set to the default of 10 days (864000 seconds).
We have  a scheduled cronjob to run repair once each week on every node, each on another day.
I'm sure that none of the nodes ever skipped running a repair.
We don't run compact on the nodes explicitly as I understand that running repair will trigger a
major compaction. I'm not entirely sure if it does so, but in any case the tombstones will be removed by a minor
compaction. So I expected that the reappearing data, which is a couple of months old in some cases, was long gone
by the time we added the node.

Can anyone think of any reason why the old data reappeared?

Stefan




--------------030505020104040105020301--