cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Haddad <jonathan.had...@gmail.com>
Subject Re: Deletes Reappeared even when nodes are not down
Date Fri, 13 Nov 2015 21:13:19 GMT
Any chance your clocks are off?  

> On Nov 13, 2015, at 1:09 PM, Peddi, Praveen <peddi@amazon.com> wrote:
> 
> Hi,
> We are using Cassandra 2.0.8, with replication factor of 3.
> 
> We are seeing a scenario where some of the rows in the table reappears even after they
are deleted. We have seen this in Prod 3 times in last 1 week and coincidentally all 3 times
on the same partition. We have confirmed that nodes didn’t go down (they were all started
on Oct 28). I do not see any resource issues (CPU is around 5%, memory < 10%, iostats looks
normal etc) on any nodes.
> 
> The only possible explanation we have so far is, one node that has data for affected
partition, being browned out for at least 15 minutes (our GC grace is 15 minutes) and came
back up but when it came back up, there were some rows in that partition that were deleted
on other two nodes. However I am not sure how to prove this is what is happening.
> 
> I am not sure if this is a bug in Cassandra (unlikely since we have not seen this issue
in last 8 months until this week). Any pointers as to how to find the root cause would be
greatly appreciated.
> 
> Thanks in advance.
> Praveen


Mime
View raw message