incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Old data coming alive after adding node
Date Tue, 06 Mar 2012 09:51:01 GMT
> After we added a fourth node, keeping RF=3, some old data appeared in the database.
What CL are you working at ? (Should not matter too much with repair working, just asking)


> We don't run compact on the nodes explicitly as I understand that running repair will
trigger a
> major compaction. I'm not entirely sure if it does so, but in any case the tombstones
will be removed by a minor
> compaction.
In 0.6.x tombstones were only purged during a major / manual compaction. Purging during minor
compaction came in during 0.7
https://github.com/apache/cassandra/blob/trunk/CHANGES.txt#L1467

> Can anyone think of any reason why the old data reappeared?
It sounds like you are doing things correctly. The complicating factor is 0.6 is so very old.



If I wanted to poke around some more I would conduct reads as CL one against nodes and see
if they return the "deleted" data or not. This would help me understand if the tombstone is
still out there. 

I would also poke around a lot in the logs to make sure repair was running as expected and
completing. If you find anything suspicious post examples. 

Finally I would ensure CL QUROUM was been used. 

Hope that helps.


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 6/03/2012, at 10:13 PM, Stefan Reek wrote:

> Hi,
> 
> We were running a 3-node cluster of cassandra 0.6.13 with RF=3.
> After we added a fourth node, keeping RF=3, some old data appeared in the database.
> As far as I understand this can only happen if nodetool repair wasn't run for more than
GCGraceSeconds.
> Our GCGraceSeconds is set to the default of 10 days (864000 seconds).
> We have  a scheduled cronjob to run repair once each week on every node, each on another
day.
> I'm sure that none of the nodes ever skipped running a repair.
> We don't run compact on the nodes explicitly as I understand that running repair will
trigger a
> major compaction. I'm not entirely sure if it does so, but in any case the tombstones
will be removed by a minor
> compaction. So I expected that the reappearing data, which is a couple of months old
in some cases, was long gone
> by the time we added the node.
> 
> Can anyone think of any reason why the old data reappeared?
> 
> Stefan


Mime
View raw message