cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peddi, Praveen" <pe...@amazon.com>
Subject Deletes Reappeared even when nodes are not down
Date Fri, 13 Nov 2015 21:09:01 GMT
Hi,
We are using Cassandra 2.0.8, with replication factor of 3.

We are seeing a scenario where some of the rows in the table reappears even after they are
deleted. We have seen this in Prod 3 times in last 1 week and coincidentally all 3 times on
the same partition. We have confirmed that nodes didn't go down (they were all started on
Oct 28). I do not see any resource issues (CPU is around 5%, memory < 10%, iostats looks
normal etc) on any nodes.

The only possible explanation we have so far is, one node that has data for affected partition,
being browned out for at least 15 minutes (our GC grace is 15 minutes) and came back up but
when it came back up, there were some rows in that partition that were deleted on other two
nodes. However I am not sure how to prove this is what is happening.

I am not sure if this is a bug in Cassandra (unlikely since we have not seen this issue in
last 8 months until this week). Any pointers as to how to find the root cause would be greatly
appreciated.

Thanks in advance.
Praveen

Mime
View raw message