cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peddi, Praveen" <pe...@amazon.com>
Subject Re: Deletes Reappeared even when nodes are not down
Date Fri, 13 Nov 2015 21:20:03 GMT
Hi Jon,
Thanks for your response.
Clock skews is not a possibility because the row that got re-appeared is 20 mins older than
the one got deleted (based on last modified date field). We are definitely not talking about
few millis here.

Praveen

From: Jon Haddad <jonathan.haddad@gmail.com<mailto:jonathan.haddad@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Friday, November 13, 2015 at 4:13 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Deletes Reappeared even when nodes are not down

Any chance your clocks are off?

On Nov 13, 2015, at 1:09 PM, Peddi, Praveen <peddi@amazon.com<mailto:peddi@amazon.com>>
wrote:

Hi,
We are using Cassandra 2.0.8, with replication factor of 3.

We are seeing a scenario where some of the rows in the table reappears even after they are
deleted. We have seen this in Prod 3 times in last 1 week and coincidentally all 3 times on
the same partition. We have confirmed that nodes didn't go down (they were all started on
Oct 28). I do not see any resource issues (CPU is around 5%, memory < 10%, iostats looks
normal etc) on any nodes.

The only possible explanation we have so far is, one node that has data for affected partition,
being browned out for at least 15 minutes (our GC grace is 15 minutes) and came back up but
when it came back up, there were some rows in that partition that were deleted on other two
nodes. However I am not sure how to prove this is what is happening.

I am not sure if this is a bug in Cassandra (unlikely since we have not seen this issue in
last 8 months until this week). Any pointers as to how to find the root cause would be greatly
appreciated.

Thanks in advance.
Praveen


Mime
View raw message