Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B36F8F042 for ; Tue, 2 Apr 2013 02:07:25 +0000 (UTC) Received: (qmail 16063 invoked by uid 500); 2 Apr 2013 02:07:23 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 16036 invoked by uid 500); 2 Apr 2013 02:07:23 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 16028 invoked by uid 99); 2 Apr 2013 02:07:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Apr 2013 02:07:22 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of goudarzi@gmail.com designates 209.85.216.53 as permitted sender) Received: from [209.85.216.53] (HELO mail-qa0-f53.google.com) (209.85.216.53) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Apr 2013 02:07:18 +0000 Received: by mail-qa0-f53.google.com with SMTP id k4so7382qaq.19 for ; Mon, 01 Apr 2013 19:06:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=97/e94v/razdFC9oLATu5EJevIlXYKPBOmQjzU9xSBw=; b=UuZK2ByWXjURgW8GaT5yr+20oNmUyKypsTT2gKH+LdRkAwt/SEQAH3qNQxuKzYewom viG0xFJ3HOKq8ECISHP/FONWDqDWl4EyRf1Tj37IzcLzV56Nwd3NwDhbyVO7fznchUTd 1yUyzMLqsDkFlMbDKI/mIXbCAouSCht4i1lpJCl8o0vEICiifm8L5ROx7mwOiCMQjhHe 4eoLi46QLty5qzuGoyf4PAHd4DsGQ1OnyfBSQx60UpSdH+Di+OJ+p0NYyrYnNtgpU4NR 0uoi6O+lU1d800/TrrPOoAbLoz5rL5H5ryT9JgsOOb3KvxsQaE/PyNlPteZHdITmlYvV VaRA== MIME-Version: 1.0 X-Received: by 10.229.61.65 with SMTP id s1mr773651qch.114.1364868417926; Mon, 01 Apr 2013 19:06:57 -0700 (PDT) Received: by 10.229.46.83 with HTTP; Mon, 1 Apr 2013 19:06:57 -0700 (PDT) In-Reply-To: References: <84B566FB5B7B244B81E6F1FEADA9087701D2A9E565@LDNPCMMGMB01.INTRANET.BARCAPINT.COM> <21BB7BBD-80FB-477D-8B02-209DC65069C9@thelastpickle.com> Date: Mon, 1 Apr 2013 19:06:57 -0700 Message-ID: Subject: Re: Lots of Deleted Rows Came back after upgrade 1.1.6 to 1.1.10 From: Arya Goudarzi To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11c2ba80373f9804d9573186 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2ba80373f9804d9573186 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Filed https://issues.apache.org/jira/browse/CASSANDRA-5411 https://issues.apache.org/jira/browse/CASSANDRA-5412 However, I don't think that was our issue as we don't have nodes down for a long period of time. The longest we had a node down was for a day, and it was replaced within few hours. I have tried very hard to reproduce this issue by putting our production snapshot on our staging cluster and run the upgrade from 1.1.6 to 1.1.10, but I was not successful. So, I proposed to upgrade our cluster in production and it happened again. Now our production includes lots of these zombies that we have to delete. Luckily our engineers already written scripts to facilitate those issues, but why why why C*? After 3 years of developing apps with C* and maintaining it, I have never been this much disappointed. Anyways, I cut my nagging, but this time before i have engineers clean up the data, I checked the timestamps of a handful of returned rows. The timestamps where from before they were deleted, so they are officially deleted rows that came back to life. We have repairs running every night, unless repair is incorrectly reporting successful repairs, I really have no clue where to start and find answers for this strange issue except medicating myself with scotch whiskey so that I can sleep at night, not having to think what C* is going to bring on my desk the next morning. -Arya On Sun, Mar 31, 2013 at 3:09 AM, aaron morton wrot= e: > But what if the gc_grace was changed to a lower value as part of a schema > migration after the hints have been marked with TTLs equal to the lower > gc_grace before the migration? > > There would be a chance then if the tombstones had been purged. > Want to raise a ticket ? > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 29/03/2013, at 2:58 AM, Arya Goudarzi wrote: > > I am not familiar with that part of the code yet. But what if the gc_grac= e > was changed to a lower value as part of a schema migration after the hint= s > have been marked with TTLs equal to the lower gc_grace before the > migration? > > From what you've described, I think this is not an issue for us as we did > not have a node down for a long period of time, but just pointing out wha= t > I think could happen based on what you've described. > > On Sun, Mar 24, 2013 at 10:03 AM, aaron morton w= rote: > >> I could imagine a scenario where a hint was replayed to a replica after >> all replicas had purged their tombstones >> >> Scratch that, the hints are TTL'd with the lowest gc_grace. >> Ticket closed https://issues.apache.org/jira/browse/CASSANDRA-5379 >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 24/03/2013, at 6:24 AM, aaron morton wrote: >> >> Beside the joke, would hinted handoff really have any role in this issue= ? >> >> I could imagine a scenario where a hint was replayed to a replica after >> all replicas had purged their tombstones. That seems like a long shot, i= t >> would need one node to be down for the write and all up for the delete a= nd >> for all of them to have purged the tombstone. But maybe we should have a >> max age on hints so it cannot happen. >> >> Created https://issues.apache.org/jira/browse/CASSANDRA-5379 >> >> Ensuring no hints are in place during an upgrade would work around. I >> tend to make sure hints and commit log are clear during an upgrade. >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 22/03/2013, at 7:54 AM, Arya Goudarzi wrote: >> >> Beside the joke, would hinted handoff really have any role in this issue= ? >> I have been struggling to reproduce this issue using the snapshot data >> taken from our cluster and following the same upgrade process from 1.1.6= to >> 1.1.10. I know snapshots only link to active SSTables. What if these >> returned rows belong to some inactive SSTables and some bug exposed itse= lf >> and marked them as active? What are the possibilities that could lead to >> this? I am eager to find our as this is blocking our upgrade. >> >> On Tue, Mar 19, 2013 at 2:11 AM, wrote: >> >>> This obscure feature of Cassandra is called =E2=80=9Chaunted handoff=E2= =80=9D.**** >>> >>> ** ** >>> >>> Happy (early) April Fools J**** >>> >>> ** ** >>> >>> *From:* aaron morton [mailto:aaron@thelastpickle.com] >>> *Sent:* Monday, March 18, 2013 7:45 PM >>> *To:* user@cassandra.apache.org >>> *Subject:* Re: Lots of Deleted Rows Came back after upgrade 1.1.6 to >>> 1.1.10**** >>> >>> ** ** >>> >>> As you see, this node thinks lots of ranges are out of sync which >>> shouldn't be the case as successful repairs where done every night prio= r to >>> the upgrade. **** >>> >>> Could this be explained by writes occurring during the upgrade process = ? >>> **** >>> >>> ** ** >>> >>> I found this bug which touches timestamp and tomstones which was fixed >>> in 1.1.10 but am not 100% sure if it could be related to this issue: >>> https://issues.apache.org/jira/browse/CASSANDRA-5153**** >>> >>> Me neither, but the issue was fixed in 1.1.0**** >>> >>> ** ** >>> >>> It appears that the repair task that I executed after upgrade, brought >>> back lots of deleted rows into life.**** >>> >>> Was it entire rows or columns in a row?**** >>> >>> Do you know if row level or column level deletes were used ? **** >>> >>> ** ** >>> >>> Can you look at the data in cassanca-cli and confirm the timestamps on >>> the columns make sense ? **** >>> >>> ** ** >>> >>> Cheers**** >>> >>> ** ** >>> >>> -----------------**** >>> >>> Aaron Morton**** >>> >>> Freelance Cassandra Consultant**** >>> >>> New Zealand**** >>> >>> ** ** >>> >>> @aaronmorton**** >>> >>> http://www.thelastpickle.com**** >>> >>> ** ** >>> >>> On 16/03/2013, at 2:31 PM, Arya Goudarzi wrote:***= * >>> >>> >>> >>> **** >>> >>> Hi,**** >>> >>> ** ** >>> >>> I have upgraded our test cluster from 1.1.6 to 1.1.10. Followed by >>> running repairs. It appears that the repair task that I executed after >>> upgrade, brought back lots of deleted rows into life. Here are some >>> logistics:**** >>> >>> ** ** >>> >>> - The upgraded cluster started from 1.1.1 -> 1.1.2 -> 1.1.5 -> 1.1.6 **= * >>> * >>> >>> - Old cluster: 4 node, C* 1.1.6 with RF3 using NetworkTopology;**** >>> >>> - Upgrade to : 1.1.10 with all other settings the same;**** >>> >>> - Successful repairs were being done on this cluster every night;**** >>> >>> - Our clients use nanosecond precision timestamp for cassandra calls;**= * >>> * >>> >>> - After upgrade, while running repair I say some log messages like this >>> in one node:**** >>> >>> ** ** >>> >>> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:54,847 >>> AntiEntropyService.java (line 1022) [repair >>> #0990f320-8da9-11e2-0000-e9b2bd8ea1bd] Endpoints /XX.194.60 and / >>> 23.20.207.56 have 2223 range(s) out of sync for App**** >>> >>> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:54,877 >>> AntiEntropyService.java (line 1022) [repair >>> #0990f320-8da9-11e2-0000-e9b2bd8ea1bd] Endpoints /XX.250.43 and / >>> 23.20.207.56 have 161 range(s) out of sync for App**** >>> >>> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:55,097 >>> AntiEntropyService.java (line 1022) [repair >>> #0990f320-8da9-11e2-0000-e9b2bd8ea1bd] Endpoints /XX.194.60 and / >>> 23.20.250.43 have 2294 range(s) out of sync for App**** >>> >>> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:59,190 >>> AntiEntropyService.java (line 789) [repair >>> #0990f320-8da9-11e2-0000-e9b2bd8ea1bd] App is fully synced (13 remainin= g >>> column family to sync for this session)**** >>> >>> ** ** >>> >>> As you see, this node thinks lots of ranges are out of sync which >>> shouldn't be the case as successful repairs where done every night prio= r to >>> the upgrade. **** >>> >>> ** ** >>> >>> The App CF uses SizeTiered with gc_grace of 10 days. It has caching =3D >>> 'ALL', and it is fairly small (11Mb on each node).**** >>> >>> ** ** >>> >>> I found this bug which touches timestamp and tomstones which was fixed >>> in 1.1.10 but am not 100% sure if it could be related to this issue: >>> https://issues.apache.org/jira/browse/CASSANDRA-5153**** >>> >>> ** ** >>> >>> Any advice on how to dig deeper into this would be appreciated.**** >>> >>> ** ** >>> >>> Thanks,**** >>> >>> -Arya**** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> >>> _______________________________________________ >>> >>> This message may contain information that is confidential or privileged= . >>> If you are not an intended recipient of this message, please delete it = and >>> any attachments, and notify the sender that you have received it in err= or. >>> Unless specifically stated in the message or otherwise indicated, you m= ay >>> not duplicate, redistribute or forward this message or any portion ther= eof, >>> including any attachments, by any means to any other person, including = any >>> retail investor or customer. This message is not a recommendation, advi= ce, >>> offer or solicitation, to buy/sell any product or service, and is not a= n >>> official confirmation of any transaction. Any opinions presented are so= lely >>> those of the author and do not necessarily represent those of Barclays. >>> This message is subject to terms available at: >>> www.barclays.com/emaildisclaimer and, if received from Barclays' Sales >>> or Trading desk, the terms available at: >>> www.barclays.com/salesandtradingdisclaimer/. By messaging with Barclays >>> you consent to the foregoing. Barclays Bank PLC is a company registered= in >>> England (number 1026167) with its registered office at 1 Churchill Plac= e, >>> London, E14 5HP. This email may relate to or be sent from other members= of >>> the Barclays group. >>> >>> _______________________________________________ >>> >> >> >> >> > > --001a11c2ba80373f9804d9573186 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Filed=C2=A0
https://issues.apache.org/jira/browse/CASSANDRA-5412

However, I don't think that was our issue as = we don't have nodes down for a long period of time. The longest we had = a node down was for a day, and it was replaced within few hours. I have tri= ed very hard to reproduce this issue by putting our production snapshot on = our staging cluster and run the upgrade from 1.1.6 to 1.1.10, but I was not= successful. So, I proposed to upgrade our cluster in production and it hap= pened again. Now our production includes lots of these zombies that we have= to delete. Luckily our engineers already written scripts to facilitate tho= se issues, but why why why C*? After 3 years of developing apps with C* and= maintaining it, I have never been this much disappointed. Anyways, I cut m= y nagging, but this time before i have engineers clean up the data, I check= ed the timestamps of a handful of returned rows. The timestamps where from = before they were deleted, so they are officially deleted rows that came bac= k to life. We have repairs running every night, unless repair is=C2=A0incor= rectly=C2=A0reporting successful repairs, I really have no clue where to st= art and find answers for this strange issue except medicating myself with s= cotch=C2=A0whiskey so that I can sleep at night, not having to think what C= * is going to bring on my desk the next morning.=C2=A0=C2=A0

-Arya


On Sun, Mar 31, 2013 at 3:09 AM, = aaron morton <aaron@thelastpickle.com> wrote:
But what if the gc_grace was changed to= a lower value as part of a schema migration after the hints have been mark= ed with TTLs equal to the lower gc_grace before the migration?=C2=A0
There would be a chance then if the tombstones had been purged.=C2=A0=
Want to raise a ticket ?=C2=A0

Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 29/03/2013, at 2:58 AM, Arya = Goudarzi <goudar= zi@gmail.com> wrote:

I am not fam= iliar with that part of the code yet. But what if the gc_grace was changed = to a lower value as part of a schema migration after the hints have been ma= rked with TTLs equal to the lower gc_grace before the migration?=C2=A0

From what you've described, I think this is not an issue= for us as we did not have a node down for a long period of time, but just = pointing out what I think could happen based on what you've described.<= br>
On Sun, Mar 24, 2013 at 10:03 AM, aaron mort= on <aaron@thelastpickle.com> wrote:
I could imagine a =C2=A0scenario where a hint w= as replayed to a replica after all replicas had purged their tombstones
Scratch that, the hints are TTL'd with the lowest gc= _grace.=C2=A0

Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 24/03/2013, at 6:24 AM, aaron morton <<= a href=3D"mailto:aaron@thelastpickle.com" target=3D"_blank">aaron@thelastpi= ckle.com> wrote:

Beside the joke, would hinted handoff really have= any role in this issue?
I could imagine a =C2=A0scenario where= a hint was replayed to a replica after all replicas had purged their tombs= tones. That seems like a long shot, it would need one node to be down for t= he write and all up for the delete and for all of them to have purged the t= ombstone. But maybe we should have a max age on hints so it cannot happen.= =C2=A0


Ensuring no hints are in place d= uring an upgrade would work around. I tend to make sure hints and commit lo= g are clear during an upgrade.=C2=A0

Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 22/03/2013, at 7:54 AM, Arya Goudarzi <goudarzi@gmail.com> wrote:
Beside the joke, would hinted handoff rea= lly have any role in this issue? I have been struggling to reproduce this i= ssue using the snapshot data taken from our cluster and following the same = upgrade process from 1.1.6 to 1.1.10. I know snapshots only link to active = SSTables. What if these returned rows belong to some inactive SSTables and = some bug exposed itself and marked them as active? What are the possibiliti= es that could lead to this? I am eager to find our as this is blocking our = upgrade.

On Tue, Mar 19, 2013 at 2:11 AM, <= moshe.kranc@barclays.com> wrote:

This obscure = feature of Cassandra is called =E2=80=9Chaunted handoff=E2=80=9D.=

=C2=A0

Happy (early) April= Fools J

=C2=A0

From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Monday, March 18, 2013 7:45 PM
To: user@cassandra.apache.org=
Subject: Re: Lots of Deleted Rows Came back after upgrade 1.1.6 = to 1.1.10

=C2=A0

As y= ou see, this node thinks lots of ranges are out of sync which shouldn't= be the case as successful repairs where done every night prior to the upgr= ade.=C2=A0

Could this be explained by writes = occurring during the upgrade process ?=C2=A0

=C2=A0

I found this bug which touches timestamp and tomston= es which was fixed in 1.1.10 but am not 100% sure if it could be related to= this issue:=C2=A0https://issues.apache.org/jira/browse/CASSANDRA-5= 153

Me neither, but the issue was fixed in = 1.1.0

=C2=A0

=C2=A0It appears that the repair task that I executed after upgrade, brough= t back lots of deleted rows into life.

Was it entire rows or columns in a row?

<= /div>

Do you know if row level or column level delete= s were used ?=C2=A0

<= /u>=C2=A0

Can you look at the d= ata in cassanca-cli and confirm the timestamps on the columns make sense ? = =C2=A0

=C2=A0

Cheers

= =C2=A0

-----------------

Aaron Morton

Freelance Cassandra Consul= tant

New Zealand

=C2=A0

@aaronmorton

=C2=A0

On 16/03/2013, at 2:31 PM, Arya Goudarzi <goudarzi@gmail.com> wrote= :



Hi,

=C2=A0

I have upgraded our test cluster from 1.= 1.6 to 1.1.10. Followed by running repairs. It appears that the repair task= that I executed after upgrade, brought back lots of deleted rows into life= . Here are some logistics:

=C2=A0

- The upgraded cluster started from 1.1.1 -> 1.1.2 ->= ; 1.1.5 -> 1.1.6=C2=A0

- Old cluster: 4 node, C* 1.1.6 with RF3 using NetworkTopology;=

- Upgrade to : 1.1.10 with all other sett= ings the same;

- Success= ful repairs were being done on this cluster every night;

- Our clients use nanosecond precision timestam= p for cassandra calls;

-= After upgrade, while running repair I say some log messages like this in o= ne node:

=C2=A0

<= p class=3D"MsoNormal">system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19= :55:54,847 AntiEntropyService.java (line 1022) [repair #0990f320-8da9-11e2-= 0000-e9b2bd8ea1bd] Endpoints /XX.194.60 and /23.20.207.56 have 2223 range(s) out of sync for Ap= p

system.log.5: INFO [AntiEntropyStage:1] 2= 013-03-15 19:55:54,877 AntiEntropyService.java (line 1022) [repair #0990f32= 0-8da9-11e2-0000-e9b2bd8ea1bd] Endpoints /XX.250.43 and /23.20.207.56 have 161 range(s) out of = sync for App

system.log.5: INFO [AntiEntropyStage:1] 2= 013-03-15 19:55:55,097 AntiEntropyService.java (line 1022) [repair #0990f32= 0-8da9-11e2-0000-e9b2bd8ea1bd] Endpoints /XX.194.60 and /23.20.250.43 have 2294 range(s) out of= sync for App

system.log.5: INFO [AntiEntropyStage:1] 2= 013-03-15 19:55:59,190 AntiEntropyService.java (line 789) [repair #0990f320= -8da9-11e2-0000-e9b2bd8ea1bd] App is fully synced (13 remaining column fami= ly to sync for this session)

=C2=A0

=

As you see, this node thinks lots of ranges are out = of sync which shouldn't be the case as successful repairs where done ev= ery night prior to the upgrade.=C2=A0

=C2=A0

The App CF uses SizeTiered with gc_grace of 10 days. It ha= s caching =3D 'ALL', and it is fairly small (11Mb on each node).=

=C2=A0

I found this bug which touches timestamp and tomstones whi= ch was fixed in 1.1.10 but am not 100% sure if it could be related to this = issue:=C2=A0https://issues.apache.org/jira/browse/CASSANDRA-5153

Thanks,

-Arya

=C2=A0

=C2=A0

=C2=A0

____________________________________= ___________

This message may contain information that is confidential= or privileged. If=20 you are not an intended recipient of this message, please delete it and any= =20 attachments, and notify the sender that you have received it in error. Unle= ss=20 specifically stated in the message or otherwise indicated, you may not=20 duplicate, redistribute or forward this message or any portion thereof,=20 including any attachments, by any means to any other person, including any= =20 retail investor or customer. This message is not a recommendation, advice, = offer=20 or solicitation, to buy/sell any product or service, and is not an official= =20 confirmation of any transaction. Any opinions presented are solely those of= the=20 author and do not necessarily represent those of Barclays. This message is= =20 subject to terms available at: www.barclays.com/emaildisclaimer=20 and, if received from Barclays' Sales or Trading desk, the terms availa= ble at:=20 www.barclays.com/salesandtradingdisclaimer/.=20 By messaging with Barclays you consent to the foregoing. Barclays Bank PLC = is a=20 company registered in England (number 1026167) with its registered office a= t 1=20 Churchill Place, London, E14 5HP. This email may relate to or be sent from = other=20 members of the Barclays group.

______________________________________= _________







--001a11c2ba80373f9804d9573186--