Or, maybe if someone has an idea of what might be behind the following oddness: columns are deleted, then about two days later (which is what we have GCGraceSeconds set to) they pop back up again, shortly after running a "nodetool repair" on all the nodes in the cluster.
One tell-tale clue is that the timestamps on the columns that the client logs show were deleted look to be the original timestamps, as they are far back in the past and our code doesn't create timestamp values in the past. A bunch of investigation work has us almost convinced that the deleted columns are popping up in large numbers after a repair is done.
I've posted a message to this list before regarding deleted columns coming back and one suggestion received was to assume it's a client-side bug, so we've done a ton of things to try to rule out various possibilities and we're left with what seems like an improbability of a basic problem on the Cassandra server.
Just to close, we're running nodetool repair nightly (right after we do a nodetool flush), we have GCGraceSeconds set to 2 days, and read/writes for our tests are CL.ALL. Should we be running nodetool compact also?