After we added a fourth node, keeping RF=3, some old data appeared in the database.
What CL are you working at ? (Should not matter too much with repair working, just asking)


We don't run compact on the nodes explicitly as I understand that running repair will trigger a
major compaction. I'm not entirely sure if it does so, but in any case the tombstones will be removed by a minor
compaction.
In 0.6.x tombstones were only purged during a major / manual compaction. Purging during minor compaction came in during 0.7
https://github.com/apache/cassandra/blob/trunk/CHANGES.txt#L1467

Can anyone think of any reason why the old data reappeared?
It sounds like you are doing things correctly. The complicating factor is 0.6 is so very old. 


If I wanted to poke around some more I would conduct reads as CL one against nodes and see if they return the "deleted" data or not. This would help me understand if the tombstone is still out there. 

I would also poke around a lot in the logs to make sure repair was running as expected and completing. If you find anything suspicious post examples. 

Finally I would ensure CL QUROUM was been used. 

Hope that helps.


-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 6/03/2012, at 10:13 PM, Stefan Reek wrote:

Hi,

We were running a 3-node cluster of cassandra 0.6.13 with RF=3.
After we added a fourth node, keeping RF=3, some old data appeared in the database.
As far as I understand this can only happen if nodetool repair wasn't run for more than GCGraceSeconds.
Our GCGraceSeconds is set to the default of 10 days (864000 seconds).
We have  a scheduled cronjob to run repair once each week on every node, each on another day.
I'm sure that none of the nodes ever skipped running a repair.
We don't run compact on the nodes explicitly as I understand that running repair will trigger a
major compaction. I'm not entirely sure if it does so, but in any case the tombstones will be removed by a minor
compaction. So I expected that the reappearing data, which is a couple of months old in some cases, was long gone
by the time we added the node.

Can anyone think of any reason why the old data reappeared?

Stefan