incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott McCarty <sc...@metajinx.com>
Subject Re: Deleted columns still coming back; CASSANDRA-{1748,1837} alive in 0.6.x?
Date Mon, 07 Feb 2011 21:45:41 GMT
Yes, we failed to run nodetool repair for quite a while and I believe it
might have been our situation that prompted the addition of that info to the
wiki :-)

We've tried/are trying two of the suggested steps there, but haven't done
the process of removing/reinserting the pseudo-failed nodes (all of them?).
 I'm thinking that won't make a difference as these deletes are being all
the time as the columns pop up and we automatically detect errors and remove
the columns (for the n'th time :-) so the timestamps should be much later
than any old & existing columns that aren't supposed to be there on the
nodes.  At least that's what I've convinced myself of!

As far as running flush before repair, I got that idea from the comments in
1748.  I'm not convinced at all it's necessary but thought it might help if
there was (still?) a problem in 0.6.10 code.

--Scott

On Mon, Feb 7, 2011 at 2:31 PM, Aaron Morton <aaron@thelastpickle.com>wrote:

> Was there a time where nodetool repair was not run frequently ?
>
> There are some steps listed here to reset issues
> around tombstones coming back to life
>
> http://wiki.apache.org/cassandra/Operations#Dealing_with_the_consequences_of_nodetool_repair_not_running_within_GCGraceSeconds
>
>
> <http://wiki.apache.org/cassandra/Operations#Dealing_with_the_consequences_of_nodetool_repair_not_running_within_GCGraceSeconds>Why
> do you run nodetool flush before the repair ?
>
> Hope that helps
> Aaron
>
>
> On 08 Feb, 2011,at 10:19 AM, Scott McCarty <scott@metajinx.com> wrote:
>
> Hi,
>
> Does anyone know if anything similar to
> https://issues.apache.org/jira/browse/CASSANDRA-1748 or
> https://issues.apache.org/jira/browse/CASSANDRA-1837 exists in 0.6.x
> releases?  Both of those bugs look like they were introduced, found, and
> fixed in 0.7, and CASSANDRA-1837 comments indicate that 0.6 tests passed but
> I wanted to double check on this because we've been seeing deleted columns
> reappear in our cluster.
>
> Or, maybe if someone has an idea of what might be behind the following
> oddness:  columns are deleted, then about two days later (which is what we
> have GCGraceSeconds set to) they pop back up again, shortly after running a
> "nodetool repair" on all the nodes in the cluster.
>
> One tell-tale clue is that the timestamps on the columns that the client
> logs show were deleted look to be the original timestamps, as they are far
> back in the past and our code doesn't create timestamp values in the past.
>  A bunch of investigation work has us almost convinced that the deleted
> columns are popping up in large numbers after a repair is done.
>
> I've posted a message to this list before regarding deleted columns coming
> back and one suggestion received was to assume it's a client-side bug, so
> we've done a ton of things to try to rule out various possibilities and
> we're left with what seems like an improbability of a basic problem on the
> Cassandra server.
>
> Just to close, we're running nodetool repair nightly (right after we do a
> nodetool flush), we have GCGraceSeconds set to 2 days, and read/writes for
> our tests are CL.ALL.  Should we be running nodetool compact also?
>
> Thanks,
>   Scott
>
>

Mime
View raw message