incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arya Goudarzi <gouda...@gmail.com>
Subject Re: Lots of Deleted Rows Came back after upgrade 1.1.6 to 1.1.10
Date Mon, 18 Mar 2013 18:51:39 GMT
Hi Aaron:

Thanks for your attention.

The cluster in question is a 4 node sandbox cluster we have that does not
have much traffic. I was able to chase down this issue on a CF that doesn't
change much.

That bug was flagged as fixed on 1.1.10.

They were row level deletes.

We use the nanosecond precision so they are something like this
: 1363379219546536704. Although this is recent for the last friday. I have
to find one with timestamp which is old and came back to life. Doing this
investigation this week and once I collect more info and reproduce from
snapshot, I'll let you know.

Cheers,
-Arya



On Mon, Mar 18, 2013 at 10:45 AM, aaron morton <aaron@thelastpickle.com>wrote:

> As you see, this node thinks lots of ranges are out of sync which
> shouldn't be the case as successful repairs where done every night prior to
> the upgrade.
>
> Could this be explained by writes occurring during the upgrade process ?
>
> I found this bug which touches timestamp and tomstones which was fixed in
> 1.1.10 but am not 100% sure if it could be related to this issue:
> https://issues.apache.org/jira/browse/CASSANDRA-5153
>
> Me neither, but the issue was fixed in 1.1.0
>
>  It appears that the repair task that I executed after upgrade, brought
> back lots of deleted rows into life.
>
> Was it entire rows or columns in a row?
> Do you know if row level or column level deletes were used ?
>
> Can you look at the data in cassanca-cli and confirm the timestamps on the
> columns make sense ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 16/03/2013, at 2:31 PM, Arya Goudarzi <goudarzi@gmail.com> wrote:
>
> Hi,
>
> I have upgraded our test cluster from 1.1.6 to 1.1.10. Followed by running
> repairs. It appears that the repair task that I executed after upgrade,
> brought back lots of deleted rows into life. Here are some logistics:
>
> - The upgraded cluster started from 1.1.1 -> 1.1.2 -> 1.1.5 -> 1.1.6
> - Old cluster: 4 node, C* 1.1.6 with RF3 using NetworkTopology;
> - Upgrade to : 1.1.10 with all other settings the same;
> - Successful repairs were being done on this cluster every night;
> - Our clients use nanosecond precision timestamp for cassandra calls;
> - After upgrade, while running repair I say some log messages like this in
> one node:
>
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:54,847
> AntiEntropyService.java (line 1022) [repair
> #0990f320-8da9-11e2-0000-e9b2bd8ea1bd] Endpoints /XX.194.60 and /
> 23.20.207.56 have 2223 range(s) out of sync for App
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:54,877
> AntiEntropyService.java (line 1022) [repair
> #0990f320-8da9-11e2-0000-e9b2bd8ea1bd] Endpoints /XX.250.43 and /
> 23.20.207.56 have 161 range(s) out of sync for App
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:55,097
> AntiEntropyService.java (line 1022) [repair
> #0990f320-8da9-11e2-0000-e9b2bd8ea1bd] Endpoints /XX.194.60 and /
> 23.20.250.43 have 2294 range(s) out of sync for App
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:59,190
> AntiEntropyService.java (line 789) [repair
> #0990f320-8da9-11e2-0000-e9b2bd8ea1bd] App is fully synced (13 remaining
> column family to sync for this session)
>
> As you see, this node thinks lots of ranges are out of sync which
> shouldn't be the case as successful repairs where done every night prior to
> the upgrade.
>
> The App CF uses SizeTiered with gc_grace of 10 days. It has caching =
> 'ALL', and it is fairly small (11Mb on each node).
>
> I found this bug which touches timestamp and tomstones which was fixed in
> 1.1.10 but am not 100% sure if it could be related to this issue:
> https://issues.apache.org/jira/browse/CASSANDRA-5153
>
> Any advice on how to dig deeper into this would be appreciated.
>
> Thanks,
> -Arya
>
>
>
>

Mime
View raw message