incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Deleted columns still coming back; CASSANDRA-{1748,1837} alive in 0.6.x?
Date Mon, 07 Feb 2011 21:31:49 GMT
Was there a time where nodetool repair was not run frequently ?

There are some steps listed here to reset issues around tombstones coming back to life 
http://wiki.apache.org/cassandra/Operations#Dealing_with_the_consequences_of_nodetool_repair_not_running_within_GCGraceSeconds

Why do you run nodetool flush before the repair ?

Hope that helps
Aaron


On 08 Feb, 2011,at 10:19 AM, Scott McCarty <scott@metajinx.com> wrote:

Hi,

Does anyone know if anything similar to https://issues.apache.org/jira/browse/CASSANDRA-1748
or https://issues.apache.org/jira/browse/CASSANDRA-1837 exists in 0.6.x releases?  Both
of those bugs look like they were introduced, found, and fixed in 0.7, and CASSANDRA-1837
comments indicate that 0.6 tests passed but I wanted to double check on this because we've
been seeing deleted columns reappear in our cluster.

Or, maybe if someone has an idea of what might be behind the following oddness:  columns
are deleted, then about two days later (which is what we have GCGraceSeconds set to) they
pop back up again, shortly after running a "nodetool repair" on all the nodes in the cluster.

One tell-tale clue is that the timestamps on the columns that the client logs show were deleted
look to be the original timestamps, as they are far back in the past and our code doesn't
create timestamp values in the past.  A bunch of investigation work has us almost convinced
that the deleted columns are popping up in large numbers after a repair is done.

I've posted a message to this list before regarding deleted columns coming back and one suggestion
received was to assume it's a client-side bug, so we've done a ton of things to try to rule
out various possibilities and we're left with what seems like an improbability of a basic
problem on the Cassandra server.

Just to close, we're running nodetool repair nightly (right after we do a nodetool flush),
we have GCGraceSeconds set to 2 days, and read/writes for our tests are CL.ALL.  Should we
be running nodetool compact also?

Thanks,
  Scott


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
    • Unnamed multipart/related (inline, None, 0 bytes)
View raw message