cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott McCarty <>
Subject Deleted columns still coming back; CASSANDRA-{1748,1837} alive in 0.6.x?
Date Mon, 07 Feb 2011 21:19:01 GMT

Does anyone know if anything similar to or exists in 0.6.x
releases?  Both of those bugs look like they were introduced, found, and
fixed in 0.7, and CASSANDRA-1837 comments indicate that 0.6 tests passed but
I wanted to double check on this because we've been seeing deleted columns
reappear in our cluster.

Or, maybe if someone has an idea of what might be behind the following
oddness:  columns are deleted, then about two days later (which is what we
have GCGraceSeconds set to) they pop back up again, shortly after running a
"nodetool repair" on all the nodes in the cluster.

One tell-tale clue is that the timestamps on the columns that the client
logs show were deleted look to be the original timestamps, as they are far
back in the past and our code doesn't create timestamp values in the past.
 A bunch of investigation work has us almost convinced that the deleted
columns are popping up in large numbers after a repair is done.

I've posted a message to this list before regarding deleted columns coming
back and one suggestion received was to assume it's a client-side bug, so
we've done a ton of things to try to rule out various possibilities and
we're left with what seems like an improbability of a basic problem on the
Cassandra server.

Just to close, we're running nodetool repair nightly (right after we do a
nodetool flush), we have GCGraceSeconds set to 2 days, and read/writes for
our tests are CL.ALL.  Should we be running nodetool compact also?


View raw message