cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Coli <rc...@eventbrite.com>
Subject Re: Repair completes successfully but data is still inconsistent
Date Wed, 19 Nov 2014 00:43:21 GMT
On Tue, Nov 18, 2014 at 12:46 PM, Michael Shuler <michael@pbandjelly.org>
wrote:

> `nodetool cleanup` also looks interesting as an option.


I don't understand why cleanup or scrub would help with a case where data
is being un-tombstoned.

"
1 November - column is deleted - gc_grace_period is 10 days
8 November - all 3 replicas have tombstone
13/14 November - column/tombstone is gone on 2 replicas, 3rd replica has
the original value (!), with the original timestamp…
"

To me the smoking gun here is that 3rd replica has the original value.

@OP : can you repro if you run a major compaction between the deletion and
the tombstone collection?

Basically, I am conjecturing that a compaction bug or one of the handful of
"unmask previously deleted data" bugs are resulting in the unmasking of a
non-tombstone row which is sitting in a SStable.

OP could also support this conjecture by running sstablekeys on other
SSTables on "3rd replica" and determining what masked values there are for
the row prior to deletion. If the data is sitting in an old SStable, this
is suggestive.

One last question for OP would be whether the nodes were restarted during
the time period this bug was observed. An assortment of the "unmask
previously deleted data" bugs come from "dead" sstables in the data
directory being marked "live" on a restart.

=Rob
http://twitter.com/rcolidba

Mime
View raw message